Storage medium usage prediction and alarm method and system based on object storage system

Abstract

The present invention belongs to the technical field of data storage, and provides a storage medium usage prediction and alarm method and system based on an object storage system, the method comprising: using a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, using a self-built training data set to train the storage medium usage prediction model according to specific rules; according to the model parameters of the encoder and the model parameters of the decoder determined in the model training process, repeatedly using the decoder to make predictions and generate a storage trend chart; inputting the data to be predicted into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing. The present invention can make more accurate predictions on the storage volume data, and can achieve more effective early warning processing based on the obtained prediction value.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to a storage medium usage prediction and alarm method and system based on an object storage system. Background Art

[0002] The existing alarm method is to collect the usage of each storage medium through the monitoring module, and compare it with the threshold set by the management module. When the usage is greater than or equal to the threshold, an alarm is triggered to prompt the user that the system has health anomalies. The existing method cannot predict future usage. When the write traffic is large, the available capacity of the storage medium will drop sharply, causing a significant drop in cluster IO performance until the system is unavailable. In addition, data balancing takes a long time, causing the cluster to be unable to be used normally for a long time, which has a significant impact on online business.

[0003] Therefore, it is necessary to provide a new storage medium usage prediction and alarm method and system based on an object storage system to solve the above problems. Summary of the invention

[0004] The present invention aims to provide a storage medium usage prediction and alarm method and system based on an object storage system to solve the problem that the prior art is unable to accurately predict and search for alarms for situations where storage medium usage is relatively large. The prior art is unable to predict future usage, especially when the write traffic is large, the available capacity of the storage medium will drop sharply, resulting in a significant drop in cluster IO performance, and even causing the system to be unavailable. The prior method takes a long time to balance data, resulting in the cluster being unable to be used normally for a long time, which has an adverse impact on online business, etc. The technical problem to be solved by the present invention is achieved through the following technical solutions.

[0005] The first aspect of the present invention proposes a storage medium usage prediction and alarm method based on an object storage system, comprising: collecting the following historical parameter data of a specified time period to establish a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type; using a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, using the training data set to train the storage medium usage prediction model according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating the data for a specified number of days as a training round; according to the model parameters of the encoding layer and the model parameters of the decoding layer determined in the model training process, repeatedly using the decoding layer for prediction to generate a storage volume trend chart; inputting the data to be predicted into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing

[0006] The second aspect of the present invention proposes a storage medium usage prediction and alarm system based on an object storage system, characterized in that it executes the storage medium usage prediction and alarm method based on the object storage system described in the first aspect of the present invention, and the storage medium usage prediction and alarm system includes: a data processing module, which collects the following historical parameter data in a specified time period to establish a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type; a model construction module, which uses a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses the training data set to train the storage medium usage prediction model according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating the data of the specified number of days as a training round; a generation module, which repeatedly uses the decoding layer to perform prediction according to the model parameters of the encoding layer and the model parameters of the decoding layer determined in the model training process to generate a storage volume trend chart; a determination module, which inputs the data to be predicted into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing.

[0007] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the storage medium usage prediction and alarm method based on the object storage system described in the first aspect of the present invention.

[0008] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for predicting and warning storage medium usage based on an object storage system as described in the first aspect of the present invention is implemented.

[0009] The embodiments of the present invention include the following advantages:

[0010] Compared with the prior art, the present invention adopts a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses a self-built training data set to train the storage medium usage prediction model according to specific rules; according to the model parameters of the encoder and the model parameters of the decoder determined in the model training process, the decoder is repeatedly used for prediction to generate a storage trend chart; the data to be predicted is input into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing, which can make a more accurate prediction of the storage data, and can realize early warning processing based on the obtained prediction value, so that users can have more time to take safeguard measures, thereby reducing the impact of insufficient storage medium usage and effectively ensuring the stability and availability of the storage system.

[0011] In addition, the first weight parameter is calculated by making the model loss calculation value smaller and the second weight parameter , that is, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface are adjusted down by the above error formula. The calculated first weight parameter and the second weight parameter The storage medium usage prediction model is brought in to continuously calculate and update the calculated value of Loss, so as to optimize the storage medium usage prediction model while continuously optimizing the generated storage trend chart, so as to obtain a more accurate storage medium usage prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of an example of a method for predicting and warning storage medium usage based on an object storage system of the present invention;

[0013] Figure 2 It is a partial flow diagram of constructing a storage medium usage prediction model in the storage medium usage prediction and alarm method based on an object storage system of the present invention;

[0014] Figure 3 It is a structural block diagram of the storage medium usage prediction and alarm system based on the object storage system of the present invention;

[0015] Figure 4 is a schematic structural diagram of an electronic device embodiment according to the present invention;

[0016] Figure 5 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] In view of the above problems, the present invention proposes a storage medium usage prediction and alarm method based on an object storage system. The method adopts a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses a self-built training data set to train the storage medium usage prediction model according to specific rules; the data to be predicted is input into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing, so as to make a more accurate prediction of the storage data. The time series model with an autoregressive attention mechanism is adopted to combine various factors and historical data information to predict the trend of storage medium usage, so as to achieve the purpose of early warning, that is, based on the obtained prediction value, early warning processing can be realized, so that users can have more time to take safeguard measures, thereby reducing the impact of insufficient storage medium usage and effectively ensuring the stability and availability of the storage system.

[0019] Example 1

[0020] Refer to the following Figure 1 and Figure 2 , the contents of the present invention will be described in detail.

[0021] Figure 1 The present invention is a flowchart of an example of a method for predicting and warning storage medium usage based on an object storage system.

[0022] like Figure 1 As shown, in step S101, the following historical parameter data of the specified time period are collected to establish a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type.

[0023] Specifically, for example, 11 types of historical parameter data including network traffic (e.g., 20G), concurrent access volume (e.g., 500 concurrent accesses), CPU utilization (e.g., 20%), memory utilization (e.g., 20%), disk usage time period (e.g., 08:00-22:00), number of write requests per second (e.g., 100), data write volume per second (e.g., 2G), disk usage (e.g., 100G), disk queue length (e.g., 2.4), disk type (e.g., solid state drive, mechanical hard drive), and disk interface type (e.g., SATA, SAS) are collected from a monitoring system or log file. The above historical parameter data for the historical period three years forward from the current time are collected, and the historical storage usage of the storage medium is marked. For example, 100 pieces of data are formed as training data for a machine learning model, that is, a training data set is established, in which the above 11 types of historical parameter data are used as model input (e.g., represented by X), and the historical storage usage of the storage medium is used as model output (i.e., label Y).

[0024] Preferably, a training data set is established after data preprocessing is performed on the collected historical parameter data.

[0025] Specifically, the data preprocessing includes missing value processing, outlier detection, and data standardization or normalization. Specifically, the missing values ​​in the data are checked and processed. For example, if there is a missing value in the number of write requests per second in the eleven parameters in a piece of data, interpolation processing is performed within the upper and lower ranges of the position of the missing value of the request number. In this way, an outlier value such as the missing number of write requests per second in the eleven parameters in this piece of data is identified and processed.

[0026] If there is an abnormal value in the number of write requests per second in a piece of data, for example, the abnormal value far exceeds the maximum value, the abnormal value in the number of requests is replaced with the maximum value. Or, if the abnormal value is far less than the minimum value, the abnormal value in the number of requests is replaced with the minimum value.

[0027] For data standardization or normalization, the data of different dimensions are converted to the same scale. Batchnormal is used to standardize the same type of data.

[0028] Taking the number of write requests per second as an example, we use all the data of the number of write requests per second to calculate the mean and standard deviation, and then standardize each piece of data:

[0029] For example, the following is a parameter array for write requests per second:

[0030] requests_per_second = np.array([100, 150, 200, 180, 170, 160]),

[0031] Next, calculate the mean and standard deviation:

[0032] mean = np.mean(requests_per_second)

[0033] Among them, mean represents the mean of the parameter array of the number of write requests per second; requests_per_second represents the parameter array of the number of write requests per second.

[0034] std = np.std(requests_per_second)

[0035] Among them, std represents the standard deviation of the parameter array of the number of write requests per second; requests_per_second represents the parameter array of the number of write requests per second.

[0036] The following expression is used to represent the normalization of data:

[0037] normalized_data = (requests_per_second - mean) / std

[0038] Among them, normalized_data represents the standardized data; requests_per_second represents the parameter array of the number of write requests per second; mean represents the mean of the parameter array of the number of write requests per second; std represents the standard deviation of the parameter array of the number of write requests per second.

[0039] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0040] Next, in step S102, a time series model with an autoregressive attention mechanism is used to construct a storage medium usage prediction model, and the storage medium usage prediction model is trained using the training data set according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating the data for a specified number of days as a training round.

[0041] Specifically, a time series model with an autoregressive attention mechanism is used to construct a storage medium usage prediction model, and the storage medium usage prediction model is trained using the training data set according to specific rules.

[0042] like Figure 2 As shown, building a storage medium usage prediction model is divided into two stages. In the first stage, the historical parameter data of a specified time is input into the encoding layer to obtain a first data feature. The specified time includes one day, two days, three days, one week or two weeks forward from the current time. Specifically, the first data feature refers to an intermediate vector represented by multiple matrix parameters obtained after the eleven input parameters are processed by the encoding layer, i.e., the Encoder layer. For example, it is a multi-dimensional vector, or a multi-dimensional multi-column matrix data.

[0043] In the second stage, the first data feature is used as a hidden state and sent to the decoding layer to predict the first predicted storage usage for one day from the current time. Specifically, the prediction result of the second stage is repeatedly input into the decoding layer, and so on, and the data of the specified number of days is calculated as a training round, and the specified number of days is in the range of 800 to 1200 days.

[0044] The following expressions are used to perform model training to determine the first matrix parameter q(Xn), the second matrix parameter k(Xn), and the third matrix parameter v(Xn) of the coding layer:

[0045] Encoder(Xn) = self_attn ... [self_attn[self_attn[q(Xn)@k(Xn)@v(Xn)]]]

[0046] Where Encoder(Xn) represents the function of the encoding layer, Xn represents the nth historical parameter data of the input encoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q(Xn), k(Xn), and v(Xn) represent the first parameter matrix, the second matrix parameter, and the third matrix parameter obtained through model training, respectively.

[0047] The following expressions are used to perform model training to determine the fourth matrix parameter q'(Xn), the fifth matrix parameter k'(Xn), and the sixth matrix parameter v'(Xn) of the decoding layer:

[0048] Decoder(Xn) = self_attn ... [self_attn[self_attn[q'(Xn)@k'(Xn)@v'(Xn)]]]

[0049] Where Decoder(Xn) represents the function of the decoding layer, Xn represents the nth data feature of the input decoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q'(Xn), k'(Xn), and v'(Xn) represent the fourth parameter matrix, the fifth parameter matrix, and the sixth parameter matrix obtained through model training.

[0050] In a specific implementation, in the first stage, the data of a certain historical day (for example, one day forward from the current time) is used as input to obtain the first data feature through the encoding layer (i.e., the Encoder layer). In the second stage, the first data feature is sent to the decoding layer (i.e., the Decoder layer) as a hidden state to predict the first storage usage of the next day of the historical day (for example, one day backward from the current time). Repeat this process, input the data of a certain historical day and the predicted first storage usage of the next day into the encoding layer (i.e., the Encoder layer), obtain the data features of these two days, and then input them into the decoding layer (i.e., the Decoder layer) to predict the storage usage of these two days, and calculate the loss of the predicted storage usage of these two days and the actual label data. Repeat the second stage prediction results into the decoding layer, and so on, and calculate 1000 days of data as a training round. For example, the training rounds are 8 times.

[0051] The mean square error loss method (ie, MSE_LOSS) is used to calculate the loss value of the storage medium usage prediction model. When the calculated loss value is less than the specified value and the training rounds are between 8 and 12, the model training is stopped to obtain a trained storage medium usage prediction model.

[0052] Optionally, when the calculated loss value is less than 0.08 and the number of training rounds is 10, the model training is stopped to obtain a trained storage medium usage prediction model.

[0053] Specifically, the following expression is used to calculate the loss value between the prediction result of the storage medium usage prediction model and the actual label, so as to optimize the model training process:

[0054] Loss = 0.5 * +

[0055] Among them, Loss represents the loss value between the prediction result of the storage medium usage prediction model and the actual label. represents the output corresponding to the collected input data, and Y_hat_n represents the output calculated by the model based on the input data; and They are the first calculation parameter and the second calculation parameter (for example, based on empirical parameters, adjusted to 1000 after testing with historical data); and Respectively represent the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in the encoding layer (i.e., the Encoder layer). Specifically, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in multiple parameter matrices such as q(Xn), k(Xn), and v(Xn) in the Encoder layer.

[0056] The first weight parameter is calculated by making the above Loss calculation value (i.e. the model loss calculation value) smaller. and the second weight parameter , that is, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface are adjusted down by the above error formula. The calculated first weight parameter and the second weight parameter The storage medium usage prediction model is brought in to continuously calculate and update the calculated value of Loss, so as to optimize the storage medium usage prediction model while continuously optimizing the generated storage trend chart, so that the calculated value of Loss is less than the specified value, so as to complete the model training and obtain the trained storage medium usage prediction model.

[0057] It should be noted that in the present invention, in order to reduce the impact of the disk type and disk interface in the input data on the global (specifically, the n parameters of the disk type and disk interface are less important than other parameters), reduce the overfitting of the model, and increase the robustness of the model. By adding the L2 regularization term and (Specifically, the weight parameters corresponding to the disk type and disk interface in the multiple parameter matrices of the coding layer, such as q(Xn), k(Xn), v(Xn), etc.) In order to make the Loss smaller, a smaller θi is needed to make the above equation valid. This θi is the weight of the disk type parameters in the multiple parameter matrices of the coding layer, such as q(Xn), k(Xn), v(Xn), etc., thereby lowering the weight of the disk type parameters in the model; similarly, this These are the weights of the disk interface parameters in multiple parameter matrices of the coding layer, such as q(Xn), k(Xn), v(Xn), etc., which can lower the weights of the disk type parameters in the model.

[0058] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0059] Next, in step S103, according to the model parameters of the encoding layer and the model parameters of the decoding layer determined in the model training process, the decoding layer is repeatedly used for prediction to generate a storage capacity trend graph.

[0060] According to the model parameters of the encoding layer and the model parameters of the decoding layer determined above, the decoding layer is repeatedly used for prediction to generate a storage capacity trend graph.

[0061] After the above training, the model parameters of Encoder and Decoder are obtained. The historical parameter data of 1000 days is used as input, input into the encoding layer, obtains the first data feature, and then sent to the decoding layer to predict the storage usage of the next day (for example, one day from the current time), and the storage usage of the next day is input into the decoding layer to predict the storage usage of two days from the current time, and the decoding layer is repeatedly used for prediction to generate a storage trend chart for a period of time in the future, that is, a storage trend chart is generated.

[0062] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0063] Next, in step S104, the data to be predicted is input into the storage medium usage prediction model to obtain a corresponding prediction value to determine whether to perform early warning processing.

[0064] Specifically, at least two of the following data to be predicted are input into the storage medium usage prediction model to obtain corresponding prediction values ​​to determine whether to perform early warning processing: network traffic, concurrent access volume, CPU utilization, memory utilization, disk usage time period, number of write requests per second, data write volume per second, disk usage, disk queue length, disk type, and disk interface type.

[0065] For example, the network traffic, concurrent access volume, and CPU utilization of the current data are input into the storage medium usage prediction model to obtain corresponding prediction values.

[0066] For example, the network traffic, concurrent access volume, CPU utilization, memory utilization, disk usage time period, and number of write requests per second of current data are input into the storage medium usage prediction model to obtain corresponding prediction values.

[0067] It should be noted that the above description is only provided as an optional example and should not be construed as a limitation to the present invention.

[0068] When the calculated storage volume is greater than the warning volume, it is determined to perform warning processing, and the warning volume includes 85% of the storage medium usage. Specifically, when the obtained predicted value is greater than 85% of the storage medium usage, it is determined to perform warning processing, such as sending a warning message to the customer who uses the current storage medium, such as a warning message to stop writing data to the current storage medium, or sending a warning message to the customer who uses the current storage medium to replace the current storage medium with another storage medium.

[0069] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0070] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0071] Compared with the prior art, the present invention adopts a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses a self-built training data set to train the storage medium usage prediction model according to specific rules; according to the model parameters of the encoder and the model parameters of the decoder determined in the model training process, the decoder is repeatedly used for prediction to generate a storage trend chart; the data to be predicted is input into the storage medium usage prediction model to obtain the corresponding prediction value, which can make a more accurate prediction of the storage data; the time series model with an autoregressive attention mechanism is adopted to combine various factors and historical data information to predict the trend of storage medium usage, so as to achieve the purpose of early warning, that is, based on the obtained prediction value, early warning processing can be achieved, so that users can have more time to take safeguard measures, thereby reducing the impact of insufficient storage medium usage and effectively ensuring the stability and availability of the storage system.

[0072] In addition, the first weight parameter is calculated by making the model loss calculation value smaller and the second weight parameter , that is, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface are adjusted down by the above error formula. The calculated first weight parameter and the second weight parameter The storage medium usage prediction model is brought in to continuously calculate and update the calculated value of Loss, so as to optimize the storage medium usage prediction model while continuously optimizing the generated storage trend chart, so as to obtain a more accurate storage medium usage prediction model.

[0073] Example 2

[0074] The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.

[0075] Figure 3 It is a structural diagram of an example of a storage medium usage prediction and alarm system based on an object storage system of the present invention.

[0076] Reference Figure 3 The second aspect of the present disclosure provides a storage medium usage prediction and alarm system 300, which adopts the storage medium usage prediction and alarm method described in the first aspect of the present disclosure.

[0077] Specifically, the storage medium usage prediction and warning system 300 includes a data processing module 310 , a model building module 320 , a generation module 330 , and a determination module 340 .

[0078] In a specific embodiment, the data processing module 310 is used to collect the following historical parameter data in a specified time period to establish a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type. The model construction module 320 uses a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses the training data set to train the storage medium usage prediction model according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating the data of the specified number of days as a training round. The generation module 330 repeatedly uses the decoding layer to perform predictions based on the model parameters of the encoding layer and the model parameters of the decoding layer determined by the model training process to generate a storage volume trend chart. The determination module 340 inputs the data to be predicted into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing.

[0079] In a specific embodiment, the storage medium usage prediction model is trained using the training data set according to specific rules, including: in a first stage, historical parameter data of a specified time is input into a coding layer to obtain a first data feature, and the specified time includes one day, two days, three days, one week or two weeks calculated forward from the current time.

[0080] In the second stage, the first data feature is used as a hidden state and sent to the decoding layer to predict the first predicted storage usage for one day from the current time; the prediction result of the second stage is repeatedly input into the decoding layer, and so on, and the data of the specified number of days is calculated as a training round, and the specified number of days is in the range of 800 to 1200 days.

[0081] According to an optional implementation manner, the model parameters of the encoding layer and the model parameters of the decoding layer determined according to the model training process are repeatedly used for prediction at the decoding layer to generate a storage capacity trend graph.

[0082] The following expressions are used to perform model training to determine the first matrix parameter q(Xn), the second matrix parameter k(Xn), and the third matrix parameter v(Xn) of the coding layer:

[0083] Encoder(Xn) = self_attn ... [self_attn[self_attn[q(Xn)@k(Xn)@v(Xn)]]]

[0084] Where Encoder(Xn) represents the function of the encoding layer, Xn represents the nth historical parameter data of the input encoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q(Xn), k(Xn), and v(Xn) represent the first parameter matrix, the second matrix parameter, and the third matrix parameter obtained through model training, respectively;

[0085] The following expressions are used to perform model training to determine the fourth matrix parameter q'(Xn), the fifth matrix parameter k'(Xn), and the sixth matrix parameter v'(Xn) of the decoding layer:

[0086] Decoder(Xn) = self_attn ... [self_attn[self_attn[q'(Xn)@k'(Xn)@v'(Xn)]]]

[0087] Where Decoder(Xn) represents the function of the decoding layer, Xn represents the nth data feature of the input decoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q'(Xn), k'(Xn), and v'(Xn) represent the fourth parameter matrix, the fifth parameter matrix, and the sixth parameter matrix obtained through model training.

[0088] According to an optional implementation, it further includes: using a mean square error loss method to calculate the loss value of the storage medium usage prediction model, and when the calculated loss value is less than a specified value and the training rounds are between 8 and 12, stopping the model training to obtain a trained storage medium usage prediction model.

[0089] According to an optional implementation, it further includes: when the calculated loss value is less than 0.08 and the number of training rounds is 10, stopping model training.

[0090] Input at least two of the following data to be predicted into the storage medium usage prediction model to obtain corresponding prediction values ​​to determine whether to perform early warning processing: network traffic, concurrent access volume, CPU utilization, memory utilization, disk usage time period, number of write requests per second, data write volume per second, disk usage, disk queue length, disk type, and disk interface type.

[0091] When the calculated storage volume is greater than the warning volume, it is determined to perform early warning processing, and the warning volume includes 85% of the storage medium usage.

[0092] The following expression is used to calculate the loss value between the prediction result of the storage medium usage prediction model and the actual label to optimize the model training process:

[0093] Loss = 0.5 * +

[0094] Among them, Loss represents the loss value between the prediction result of the storage medium usage prediction model and the actual label. represents the output corresponding to the collected input data, and Y_hat_n represents the output calculated by the model based on the input data; and are the first calculation parameter and the second calculation parameter respectively; and Respectively represent the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in the encoding layer. Specifically, they are the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in multiple parameter matrices such as q(Xn), k(Xn), and v(Xn) in the Encoder layer.

[0095] The first weight parameter is calculated by making the above Loss calculation value (i.e. the model loss calculation value) smaller. and the second weight parameter , that is, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface are adjusted down by the above error formula. The calculated first weight parameter and the second weight parameter The storage medium usage prediction model is brought in to continuously calculate and update the calculated value of Loss, so as to optimize the storage medium usage prediction model while continuously optimizing the generated storage trend chart, so that the calculated value of Loss is less than the specified value, so as to complete the model training and obtain the trained storage medium usage prediction model.

[0096] It should be noted that the storage medium usage prediction and alarm method executed by the storage medium usage prediction and alarm system in this embodiment is substantially the same as the storage medium usage prediction and alarm method in Embodiment 1, and therefore, the description of the same parts is omitted.

[0097] Compared with the prior art, the present invention adopts a time series model with an autoregressive attention mechanism to construct a storage medium usage prediction model, and uses a self-built training data set to train the storage medium usage prediction model according to specific rules; according to the model parameters of the encoder and the model parameters of the decoder determined in the model training process, the decoder is repeatedly used for prediction to generate a storage trend chart; the data to be predicted is input into the storage medium usage prediction model to obtain the corresponding prediction value, which can make a more accurate prediction of the storage data; the time series model with an autoregressive attention mechanism is adopted to combine various factors and historical data information to predict the trend of storage medium usage, so as to achieve the purpose of early warning, that is, based on the obtained prediction value, early warning processing can be achieved, so that users can have more time to take safeguard measures, thereby reducing the impact of insufficient storage medium usage and effectively ensuring the stability and availability of the storage system.

[0098] In addition, the first weight parameter is calculated by making the model loss calculation value smaller and the second weight parameter , that is, the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface are adjusted down by the above error formula. The calculated first weight parameter and the second weight parameter The storage medium usage prediction model is brought in to continuously calculate and update the calculated value of Loss, so as to optimize the storage medium usage prediction model while continuously optimizing the generated storage trend chart, so as to obtain a more accurate storage medium usage prediction model.

[0099] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0100] like Figure 4 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0101] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0102] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0103] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0104] It should be understood that Figure 4 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0105] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 5 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0106] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0107] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0108] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0109] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0110] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0111] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0112] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A storage medium usage prediction and alarm method based on an object storage system, characterized in that: include: Collect the following historical parameter data in a specified time period to build a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type; A time series model with an autoregressive attention mechanism is used to construct a storage medium usage prediction model, and the storage medium usage prediction model is trained using the training data set according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating data for a specified number of days as a training round; According to the model parameters of the encoding layer and the model parameters of the decoding layer determined in the model training process, the decoding layer is repeatedly used for prediction to generate a storage capacity trend graph; The data to be predicted is input into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning process; wherein, In the first stage, historical parameter data of a specified time is input into the encoding layer to obtain a first data feature, wherein the specified time includes one day, two days, three days, one week or two weeks forward from the current time; In the second stage, the first data feature is used as a hidden state and sent to the decoding layer to predict the first predicted storage usage for one day from the current time; the prediction result of the second stage is repeatedly input into the decoding layer, and so on, and the data of the specified number of days is calculated as a training round, and the specified number of days is in the range of 800 to 1200 days; The following expressions are used to perform model training to determine the first matrix parameter q(Xn), the second matrix parameter k(Xn), and the third matrix parameter v(Xn) of the coding layer: Encoder(Xn)=self_attn...[self_attn[self_attn[q(Xn)@k(Xn)@v(Xn)]]]; Where Encoder(Xn) represents the function of the encoding layer, Xn represents the nth historical parameter data of the input encoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q(Xn), k(Xn), and v(Xn) represent the first parameter matrix, the second matrix parameter, and the third matrix parameter obtained through model training, respectively; The following expressions are used to perform model training to determine the fourth matrix parameter q'(Xn), the fifth matrix parameter k'(Xn), and the sixth matrix parameter v'(Xn) of the decoding layer: Decoder(Xn) = self_attn ... [self_attn[self_attn[q'(Xn)@k'(Xn)@v'(Xn)]]]; Where Decoder(Xn) represents the function of the decoding layer, Xn represents the nth data feature of the input decoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q'(Xn), k'(Xn), and v'(Xn) represent the fourth parameter matrix, the fifth parameter matrix, and the sixth parameter matrix obtained through model training; The following expression is used to calculate the loss value between the prediction result of the storage medium usage prediction model and the actual label to optimize the model training process: Loss = 0.5 * + ; Among them, Loss represents the loss value between the prediction result of the storage medium usage prediction model and the actual label. Indicates the output corresponding to the collected input data Represents the output calculated by the model based on the input data; and are the first calculation parameter and the second calculation parameter respectively; and Respectively represent the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in the encoding layer.

2. The storage medium usage prediction and alarm method based on the object storage system according to claim 1 is characterized in that: Further including: A mean square error loss method is used to calculate the loss value of the storage medium usage prediction model. When the calculated loss value is less than a specified value and the number of training rounds is between 8 and 12, the model training is stopped to obtain a trained storage medium usage prediction model.

3. The storage medium usage prediction and alarm method based on the object storage system according to claim 2 is characterized in that: Further including: When the calculated loss value is less than 0.08 and the training epoch is 10, stop model training.

4. The storage medium usage prediction and alarm method based on the object storage system according to claim 2 is characterized in that: include: Input at least two of the following data to be predicted into the storage medium usage prediction model to obtain corresponding prediction values ​​to determine whether to perform early warning processing: network traffic, concurrent access volume, CPU utilization, memory utilization, disk usage time period, number of write requests per second, data write volume per second, disk usage, disk queue length, disk type, and disk interface type; When the calculated storage volume is greater than the warning volume, it is determined to perform early warning processing, and the warning volume includes 85% of the storage medium usage.

5. A storage medium usage prediction and alarm system based on an object storage system, characterized in that: The method for predicting and warning storage medium usage based on an object storage system according to any one of claims 1 to 4 is implemented, and the storage medium usage prediction and warning system comprises: The data processing module is used to collect the following historical parameter data in a specified time period to establish a training data set: network traffic, concurrent access volume, memory utilization, disk usage time period, number of write requests per second, write volume per second, disk usage, disk queue length, disk type, and disk interface type; A model building module, using a time series model with an autoregressive attention mechanism to build a storage medium usage prediction model, using the training data set to train the storage medium usage prediction model according to specific rules to obtain a trained storage medium usage prediction model; the specific rules include calculating the data of a specified number of days as a training round; A generation module, which generates a storage capacity trend graph by repeatedly using the decoding layer for prediction according to the model parameters of the encoding layer and the model parameters of the decoding layer determined in the model training process; The determination module inputs the data to be predicted into the storage medium usage prediction model to obtain the corresponding prediction value to determine the early warning processing; wherein, In the first stage, historical parameter data of a specified time is input into the encoding layer to obtain a first data feature, wherein the specified time includes one day, two days, three days, one week or two weeks forward from the current time; In the second stage, the first data feature is used as a hidden state and sent to the decoding layer to predict the first predicted storage usage for one day from the current time; the prediction result of the second stage is repeatedly input into the decoding layer, and so on, and the data of the specified number of days is calculated as a training round, and the specified number of days is in the range of 800 to 1200 days; The following expressions are used to perform model training to determine the first matrix parameter q(Xn), the second matrix parameter k(Xn), and the third matrix parameter v(Xn) of the coding layer: Encoder(Xn)=self_attn...[self_attn[self_attn[q(Xn)@k(Xn)@v(Xn)]]]; Where Encoder(Xn) represents the function of the encoding layer, Xn represents the nth historical parameter data of the input encoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q(Xn), k(Xn), and v(Xn) represent the first parameter matrix, the second matrix parameter, and the third matrix parameter obtained through model training, respectively; The following expressions are used to perform model training to determine the fourth matrix parameter q'(Xn), the fifth matrix parameter k'(Xn), and the sixth matrix parameter v'(Xn) of the decoding layer: Decoder(Xn) = self_attn ... [self_attn[self_attn[q'(Xn)@k'(Xn)@v'(Xn)]]]; Where Decoder(Xn) represents the function of the decoding layer, Xn represents the nth data feature of the input decoding layer, and n is a positive integer; self_attn[ ] represents the self-attention mechanism, q'(Xn), k'(Xn), and v'(Xn) represent the fourth parameter matrix, the fifth parameter matrix, and the sixth parameter matrix obtained through model training; The following expression is used to calculate the loss value between the prediction result of the storage medium usage prediction model and the actual label to optimize the model training process: Loss = 0.5 * + ; Among them, Loss represents the loss value between the prediction result of the storage medium usage prediction model and the actual label. Indicates the output corresponding to the collected input data Represents the output calculated by the model based on the input data; and are the first calculation parameter and the second calculation parameter respectively; and Respectively represent the first weight parameter and the second weight parameter corresponding to the disk type and the disk interface in the encoding layer.

Images