Hard disk pool usage amount prediction method and server

By pre-training the timing characterization extractor of hard disk pool usage using offline data in a distributed storage system, the problem of prediction model training complexity and instability caused by insufficient data in the early stage is solved, and the accuracy of hard disk pool usage prediction is improved and response speed.

CN118940038APending Publication Date: 2024-11-12XFUSION DIGITAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411026850.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the early stages of distributed storage systems, due to the small amount of data, the hard disk pool usage prediction model based on machine learning/deep learning is difficult to fully learn the complex timing characteristics of the data, affecting the prediction accuracy.

Method used

Pre-training of feature extractors through offline data, a trained timing characterization extractor is obtained, and it is directly applied to predictive model training in the online stage, reducing the complexity and time overhead of feature extraction, and improving the response speed and stability of model training. sex.

Benefits of technology

It significantly shortens the prediction model training time, improves the response speed and timeliness of model training, enhances the stability and reliability of prediction results, and reduces the fluctuations in model performance caused by data fluctuations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940038A_ABST
    Figure CN118940038A_ABST
Patent Text Reader

Abstract

The invention provides a hard disk pool usage amount prediction method, and the method comprises a training stage and a prediction stage: the training stage comprises the steps: obtaining offline data, carrying out the training of a representation extractor based on the offline data, and obtaining a trained representation extractor; obtaining online data, extracting the online data by using the trained representation extractor to obtain time sequence features, and training a prediction model based on the time sequence features to obtain a trained prediction model; the prediction stage comprises the following steps: obtaining prediction data; based on the trained representation extractor, time sequence representation corresponding to the prediction data is extracted; and inputting the time sequence representation corresponding to the prediction data into the trained prediction model to obtain a prediction result. Therefore, the time for training the prediction model by using the online data is remarkably shortened, the complexity and time overhead of prediction model training are greatly reduced, and the response speed and timeliness of model training are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and server for predicting hard disk pool usage. Background Art

[0002] Distributed storage systems play a core role in modern data management, especially in monitoring the usage of hard disk pools. These systems can periodically collect the user's hard disk pool usage data and display the dynamic changes of this data in the form of a curve chart on the system page. Such a visual display allows users to intuitively observe the changes in storage pool usage and decide whether to expand the storage system according to actual needs.

[0003] With the rapid development of artificial intelligence technology, time series prediction methods based on machine learning and deep learning are gradually being applied in the field of disk pool usage prediction. The core idea of ​​this method is to analyze historical data, dig out the potential laws and features, and then build a model that can predict future trends. The amount of data directly determines the richness of information that the model can learn. When the amount of data is large enough, the model can understand the inherent laws of the data more comprehensively and deeply, thereby improving the accuracy of the prediction. Therefore, in order to achieve a high prediction accuracy, a large amount of historical data is usually required as the basis for model training.

[0004] The real-time data collection feature of distributed storage systems makes data generation and collection a continuous process. In the early stages of system deployment, due to the relatively short running time and the small amount of data that can be collected, the prediction model based on machine learning / deep learning is difficult to fully learn the complex time series characteristics of the data, thus affecting the accuracy of the model prediction. Summary of the invention

[0005] The embodiments of the present application provide a hard disk pool usage prediction method and server, aiming to improve the accuracy of model prediction.

[0006] The technical solutions provided by the embodiments of this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for predicting hard disk pool usage, the method comprising a training phase and a prediction phase:

[0008] The training phase includes:

[0009] Acquire offline data, and train a representation extractor based on the offline data to obtain a trained representation extractor;

[0010] Acquire online data, and use the trained representation extractor to extract the online data to obtain time series features, and train a prediction model based on the time series features to obtain a trained prediction model;

[0011] The prediction stage includes: acquiring prediction data, the prediction data including hard disk pool usage time series data for predicting hard disk pool usage; extracting the time series representation corresponding to the prediction data based on the trained representation extractor; inputting the time series representation corresponding to the prediction data into the trained prediction model to obtain a prediction result, and the prediction result is used to represent the size of the hard disk pool usage.

[0012] The method provided by the embodiment of the present application includes a training phase and a prediction phase. In the training phase, offline data is obtained and the time series representation extractor of the hard disk pool usage is pre-trained to obtain a trained time series representation extractor. Based on the time series representation extractor trained in the offline phase, the time series features of the online data are extracted, and then based on the extracted time series representation, a prediction model for predicting the size of the hard disk pool usage is trained. After the feature extractor is trained, it can be directly applied to the training process of the prediction model. When the online data is obtained, the trained feature extractor can be directly applied to perform feature conversion, and these features are then used to train the prediction model. Compared with the prediction model itself learning and extracting useful time series features from the obtained online data, and then training the model based on the extracted time series features, the method provided by the embodiment of the present application significantly shortens the time for training the prediction model using online data, greatly reduces the complexity and time overhead of the prediction model training, and improves the response speed and timeliness of the model training. In addition, the pre-trained feature extractor has good generalization ability for multiple data modes after training with offline data. Applying it in the training process of the prediction model can reduce the model performance fluctuations caused by data fluctuations and enhance the stability and reliability of the prediction results.

[0013] In a possible implementation, the training of the representation extractor based on the offline data to obtain a trained representation extractor includes:

[0014] training a representation extractor using the offline data;

[0015] Extracting the time series representation of the offline data based on the time series representation extractor;

[0016] Training a prediction model using the time series representation of the offline data;

[0017] Evaluate the trained prediction model to obtain evaluation results;

[0018] When the evaluation result meets the first preset requirement, stop training the representation extractor; when the evaluation result does not meet the first preset requirement, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the evaluation result meets the first preset requirement.

[0019] In a possible implementation, the using the offline data to train the representation extractor includes:

[0020] A representation extractor is trained based on the time series self-supervised learning algorithm TS2Vec and the offline data;

[0021] When the evaluation result does not meet the first preset requirement, updating the model parameters of the representation extractor, and continuing to train the time series representation extractor using the offline data until the evaluation result meets the first preset requirement, includes:

[0022] When the evaluation result does not meet the first preset requirement, the hyperparameters of the TS2Vec algorithm are updated based on the evaluation result, and the training representation extractor based on the TS2Vec algorithm and the offline data continues until the evaluation result meets the first preset requirement.

[0023] In a possible implementation, the evaluation result includes a prediction error, and the evaluation of the trained prediction model to obtain the evaluation result includes:

[0024] Based on the prediction model, the usage of the hard disk pool at each time point in the evaluation data set is predicted to obtain the prediction result;

[0025] Calculating a prediction error of the prediction result;

[0026] When the evaluation result meets the first preset requirement, the training of the representation extractor is stopped; when the evaluation result does not meet the first preset requirement, the model parameters of the representation extractor are updated, and the training of the time series representation extractor using the offline data continues until the evaluation result meets the first preset requirement, including:

[0027] When the prediction error is less than a first threshold, stop training the representation extractor; when the prediction error is not less than the first threshold, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the prediction error is less than the first threshold.

[0028] In a possible implementation, the prediction error includes a weighted sum of a root mean square error and a mean absolute error of the prediction result, and calculating the prediction error of the prediction result includes:

[0029] Calculating the root mean square error and mean absolute error of the prediction results;

[0030] Based on the preset weights, a weighted sum of the root mean square error and the mean absolute error of the prediction result is calculated.

[0031] In a possible implementation, the step of training a prediction model using the time series representation of the offline data includes:

[0032] Based on the time series representation of the offline data, using a ridge regression algorithm to select different ridge regression parameters to train a prediction model, to obtain a first preset number of prediction models;

[0033] The step of evaluating the trained prediction model to obtain an evaluation result includes:

[0034] Calculating the prediction error of each prediction model in the first preset number of prediction models;

[0035] Obtain the minimum prediction error value in the calculation results as the evaluation result;

[0036] When the evaluation result meets the first preset requirement, the training of the representation extractor is stopped; when the evaluation result does not meet the first preset requirement, the model parameters of the representation extractor are updated, and the training of the time series representation extractor using the offline data continues until the evaluation result meets the first preset requirement, including:

[0037] When the minimum prediction error value in the calculated results is less than a first threshold, stop training the representation extractor; when the minimum prediction error value in the calculated results is not less than the first threshold, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the prediction error is less than the first threshold.

[0038] In a possible implementation, the training of the prediction model based on the time series feature to obtain the trained prediction model includes:

[0039] A prediction model is obtained based on the time series representation training of the online data;

[0040] The trained prediction model is evaluated to determine a prediction model that meets the second preset requirement, and the prediction model that meets the second preset requirement is used to predict the prediction data.

[0041] In a possible implementation, the training of obtaining a prediction model based on the time series representation of the online data includes:

[0042] Using the online data and a second preset number of linear regression prediction algorithms to train a prediction model, a first preset number of prediction models corresponding to each linear regression prediction algorithm is obtained;

[0043] Optimizing the prediction models corresponding to the linear regression prediction algorithms based on the optimization algorithm, and continuing to train the prediction models using the online data and the first preset number of linear regression prediction algorithms until the number of optimizations reaches a third preset number;

[0044] The step of evaluating the trained prediction model to determine a prediction model that meets the second preset requirement includes:

[0045] The prediction errors of the prediction models corresponding to the various linear regression prediction algorithms are calculated, and the prediction model with the smallest prediction error is determined. The prediction model with the smallest prediction error is used to predict the prediction data.

[0046] In a possible implementation, obtaining prediction data includes:

[0047] In response to the prediction trigger instruction, and / or the storage amount of the hard disk pool usage time series data meets the prediction start condition, obtaining the hard disk pool usage time series data for predicting the hard disk pool usage;

[0048] The hard disk pool usage time series data used for predicting the hard disk pool usage is preprocessed to obtain the prediction data.

[0049] A second aspect of an embodiment of the present application provides a server, including a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the hard disk pool usage prediction method as described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for predicting hard disk pool usage provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a distributed storage system provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of another distributed storage system provided in an embodiment of the present application;

[0054] Figure 4A schematic diagram of the offline training phase flow provided in an embodiment of the present application;

[0055] Figure 5 The offline data set generation process provided in the embodiment of the present application;

[0056] Figure 6 A schematic diagram of the online training phase flow provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the online prediction phase flow provided by an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the request prediction data flow provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] It should be noted that the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0060] In order to make the following embodiments clear, the technical terms involved in this application are first introduced.

[0061] A disk pool is a storage unit composed of multiple disks, usually used to build a large-capacity, high-reliability data storage system. These disks can be combined through software or hardware to form a logical storage unit to meet the needs of large-scale data storage and management.

[0062] Time series forecasting is the process of estimating or predicting values ​​at future points or time periods based on a series of chronologically ordered data points. This type of forecasting usually involves analyzing trends, periodicity, and randomness in time series data and using this information to infer future values.

[0063] Deep learning is a branch of machine learning that aims to mimic the structure and function of the human brain and learn and model data through multi-layer neural networks. It can automatically learn abstract feature representations from data and use them for various tasks such as classification, regression, clustering, and generation. In machine learning and deep learning, "representation" usually refers to feature extraction or feature learning of data, thereby converting raw data into a more useful representation. "Representation" emphasizes converting data into a form that is easier to understand and process, so as to facilitate subsequent analysis, modeling, or decision-making.

[0064] Time series self-supervised learning: The model uses the time series data itself for self-supervised learning, that is, using unlabeled data as training samples. This means that the model does not require external labels to guide the learning process, but instead uses the inherent structure of the time series data for learning.

[0065] Time series contrastive learning: During the learning process, the model learns by comparing the similarities or differences between different time points or different samples in the time series data. This can be done by constructing positive and negative sample pairs, and then making the similarity between positive samples higher than the similarity between negative samples, so that the model can learn more discriminative feature representations.

[0066] Distributed storage systems usually have the function of monitoring the usage of hard disk pools. They periodically collect the user's hard disk pool usage data and draw a curve on the system page. Users can decide whether to expand the storage system by observing the usage change curve of the storage pool. Distributed storage systems can use time series prediction methods to predict the usage of hard disk pools in the future, providing users with auxiliary decisions to expand the storage pool size.

[0067] The embodiment of the present application proposes to collect the hard disk pool usage monitoring data of the distributed storage system offline, pre-train the hard disk pool usage time series representation extractor, and obtain a time series representation extractor that meets the preset standards. Based on the time series representation extractor trained in the offline stage, the time series representation is extracted from the acquired hard disk pool usage data, and then the hard disk pool usage prediction model is trained based on these time series representations. After the feature extractor is trained, it can be used directly in the online training stage, and the real-time or near real-time prediction data is input into this pre-trained feature extractor and quickly converted into effective features, which greatly reduces the complexity and time overhead of the online training process.

[0068] Reduced computing resource requirements: The online training phase avoids repeated training of the feature extractor, which means that a large amount of computing resources do not need to be allocated to the feature learning process, thereby reducing hardware costs and energy consumption. This is especially important for production environments with limited resources or that require fast response.

[0069] Accelerate model training: When online data for predictive model training is obtained, the pre-trained feature extractor can be directly applied to perform feature conversion on the online data, and the features corresponding to these online data can then be used to train the predictive model. Compared with the model itself learning and extracting useful time series features from the obtained online data, and then training the model based on these time series features, the method provided in the embodiment of the present application significantly shortens the process of training the predictive model using online data, and improves the response speed and timeliness of model training.

[0070] Enhanced model stability: The pre-trained feature extractor has been trained with a large amount of data and has good generalization capabilities for various data patterns. Applying it to the prediction model training process can reduce model performance fluctuations caused by data fluctuations and enhance the stability and reliability of predictions.

[0071] Simplified maintenance and iteration: As an independent product obtained through offline data training, the optimization and upgrade of the feature extractor will not affect the entire prediction process. If the feature extractor needs to be optimized, the optimized feature extractor can be replaced in the prediction model training process without adjusting other parts of the prediction model, which simplifies maintenance and iteration.

[0072] The following is an example to illustrate a model training method provided by the present application. Figure 1 ,Should Figure 1 A flowchart of a hard disk pool usage prediction method provided in an embodiment of the present application specifically includes a training phase and a prediction phase, wherein the training phase includes an offline training phase and an online training phase, and the prediction phase can also be called an online prediction phase.

[0073] In the offline training phase, i.e., the development phase of the distributed storage system, the computing device trains the time series representation extractor based on offline data using a time series self-supervised contrastive learning algorithm to obtain a trained time series representation extractor.

[0074] The offline data is the historical monitoring data of the hard disk pool usage collected from multiple distributed storage systems, and is used to train the representation extractor. Exemplarily, at least 365 days of historical monitoring data of the hard disk pool usage is collected from each distributed storage system in the multiple distributed storage systems to obtain the offline data used to train the representation extractor.

[0075] In the online training phase, that is, the phase when the distributed storage system is operating normally, the training module periodically performs the online training task of the model, extracts features and optimizes the prediction model based on the real-time or near-line hard disk pool usage data. In this process, the distributed storage system can periodically collect the user's online hard disk pool usage data, and extract the time series representation of the hard disk pool usage data based on the time series representation extractor trained in the offline phase. Based on these extracted time series representations, multiple linear regression hard disk pool usage prediction models are trained online and in parallel, and the optimal time series prediction model is selected from the multiple models obtained by training. Among them, the online data is the hard disk pool usage time series data collected when the distributed storage system is operating normally, which reflects the real-time usage of the hard disk pool and is used to train the prediction model according to the real-time usage of the hard disk pool.

[0076] In the online prediction phase, that is, during the normal operation of the distributed storage system, the prediction module predicts the user's hard disk pool usage data for a period of time in the future. This phase predicts the user's hard disk pool usage data for a period of time in the future based on the hard disk pool usage time series representation extractor obtained in the offline training phase, the time series prediction model and prediction data obtained in the online training phase, and stores the prediction results in the cache module. When the front-end queries, the predicted results are directly obtained from the cache. If there are none, the real-time prediction is triggered. Among them, the prediction data includes the hard disk pool usage time series data used to predict the hard disk pool usage. The prediction result is obtained by combining the hard disk pool usage time series data currently collected in real time, and accurately predicting the user's future hard disk pool usage, which is used to represent the size of the hard disk pool usage.

[0077] For example, when the time series data of the disk pool usage in the last seven days before the current prediction date is obtained as prediction data, the disk pool usage on the day after the current prediction date is determined as the prediction result based on the trained feature extractor and prediction model. Assuming that the current prediction date is July 26, the time series data of the disk pool usage for seven days from July 26 to July 20 is obtained, and the prediction result is the disk pool usage on July 27, for example, 185GB.

[0078] In one possible implementation, see Figure 2 , Figure 2 A schematic diagram of a distributed storage system provided for an embodiment of the present application, in which the backend of the distributed storage system may include three modules: a training module, a prediction module, and a cache module. The training module, the prediction module, and the cache module can be flexibly scheduled to execute on any suitable computing device according to the actual situation of the system resources. The computing device can provide the necessary computing resources (such as CPU, memory, GPU) so that each module can handle complex machine learning tasks, that is, perform online training, online prediction, and result storage tasks. The distributed storage system can dynamically and flexibly schedule these modules to execute on one or several available computing devices according to the current resource load and task requirements, making full use of resources and achieving high efficiency and elastic expansion.

[0079] like Figure 3 As shown, Figure 3A schematic diagram of a distributed storage system provided for an embodiment of the present application, in which a user submits a request through a front-end page, such as a data query, a model training task, or a prediction request, and sends it to the corresponding computing device at the back end of the distributed storage system. In actual application scenarios, which computing device at the back end executes the subsequent process can be adaptively set and adjusted according to actual needs. In addition, the computing devices in the distributed storage system can interact with each other to obtain the data required for training and prediction. The training module pulls real-time hard disk pool usage data from the corresponding computing device in the distributed storage system to perform feature extraction and model training optimization; the prediction module also obtains the latest hard disk pool usage data from the corresponding computing device in the distributed storage system to perform real-time prediction. After the prediction result is calculated by the prediction module, it is written back to the cache module in the distributed storage system. When the front-end service or user queries, the prediction result can be quickly obtained from the cache module, reducing real-time computing overhead and improving user experience.

[0080] In one possible implementation, when training a model online, multiple computing devices can be used to train different parts of the model in parallel. Similarly, during the online prediction phase, multiple devices can be used to process different prediction requests simultaneously. The system ensures data synchronization and coordination through network and inter-computing device communication.

[0081] Figure 4 A schematic diagram of the offline training phase flow provided in an embodiment of the present application, wherein the offline training phase can be performed by at least one computing device independent of the distributed storage system, or by at least one available computing device in the distributed storage system, wherein the computing device has the ability to collect data, process and train the data, wherein the representation extractor can be expressed as a time series representation extractor, and the offline data can be expressed as an offline data set, such as Figure 4 As shown, the offline training phase includes:

[0082] S201, obtaining hard disk pool usage time series data.

[0083] In this stage, at least one computing device can periodically collect the usage data of the hard disk pool. The computing device used for collection can use the data collection component carried by itself to perform the collection action. The collected usage time series data can include key indicators such as timestamp, hard disk capacity, used space, and remaining space. These data are usually presented in the form of time series, reflecting the usage of the hard disk pool over time. In the following, the hard disk pool usage time series data is uniformly referred to as the hard disk pool usage data.

[0084] S202: Obtain time series data on hard disk pool usage collected by multiple distributed storage systems.

[0085] The computing device collects disk pool usage time series data from multiple different distributed storage systems. These data come from different systems and different configurations. The disk pool usage data under different environments and load conditions is collected to build a comprehensive and diverse data set. This ensures that the trained model can be generalized to various practical application scenarios.

[0086] S203: Preprocess the hard disk pool usage time series data to obtain an offline data set.

[0087] The preprocessing method may include at least one of downsampling, unit conversion, missing value filling and data merging. The preprocessing method in the offline stage is the same as that in the online stage, and the online stage will not be described in detail.

[0088] Among them, because the periodic interval of the distributed storage system to collect disk pool usage data is different from the predicted time interval, the data can be downsampled. For example, if the predicted disk pool usage data interval is days, the disk pool usage data at the last moment of each day is sampled.

[0089] The underlying unit of a distributed storage system is usually bytes, and a distributed storage system usually views data in GB. Therefore, the data can be converted to convert the unit of the disk pool usage time series data to GB.

[0090] The sampled and converted data may have abnormal missing data at certain times, and the missing data can be filled. In this method, the disk pool usage value at the moment before the missing value is used as the value at the missing time.

[0091] Data integration: Multiple sets of hard disk pool usage time series data are integrated into one dataset for subsequent self-supervised pre-training of the hard disk pool usage time series representation extractor.

[0092] S204: Obtain a time series representation extractor based on offline data set training.

[0093] That is, based on the hard disk pool usage time series dataset and the time series self-supervised comparative learning algorithm TS2Vec, the time series representation extractor is self-supervised pre-trained to obtain the time series representation extractor.

[0094] The time series representation extractor may be a TCN model. In an exemplary implementation, a TCN model for time series representation extraction may be constructed before training the time series representation extractor. TCN is a neural network structure suitable for processing sequence data, which captures local and global dependencies in a sequence through convolution operations. The parameters of the model, such as the size, number, and step length of the convolution kernel, are set to adapt to the characteristics of the hard disk pool usage data.

[0095] The TS2Vec algorithm is applied to the preprocessed hard disk pool usage data to perform self-supervised pre-training on the TCN model. TS2Vec learns the temporal representation of data by constructing positive and negative sample pairs. Specifically, continuous segments can be extracted from the time series data as positive sample pairs, while randomly sampling segments from different time series or different positions as negative sample pairs. During the training process, the model will learn to distinguish between positive and negative sample pairs, thereby extracting the temporal characteristics of hard disk pool usage. By optimizing the model parameters, the representation similarity of the positive sample pairs is made as high as possible, while the representation similarity of the negative sample pairs is made as low as possible.

[0096] Therefore, TS2Vec establishes connections between sequence fragments at different time points through a contrastive learning mechanism, and learns representations that can capture the intrinsic patterns and dynamic changes of time series. This helps the model understand the laws of hard disk usage behavior, such as periodicity, trends, and abnormal changes.

[0097] S205: Determine the time series representation corresponding to the offline data set based on the time series representation extractor.

[0098] Based on the time series representation extractor obtained in step S24, the time series representation of the hard disk pool usage time series data set in step S23 is calculated.

[0099] Apply the time series representation extractor to the data at each moment of the disk pool usage time series data set. The time series representation extractor analyzes the input time series data and generates the corresponding time series representation based on its internal structure and parameters. That is, for the disk pool usage data at each moment, the time series representation extractor generates a corresponding time series representation.

[0100] Therefore, the disk pool usage time series data set is converted into corresponding time series representations, which provides key input features for the subsequent training of the prediction model. These time series representations contain useful information in the original time series data and can effectively improve the performance of the prediction model.

[0101] S206: Obtain a prediction model based on time series feature training.

[0102] Based on the time series representation and using the deep learning algorithm, a prediction model is trained, that is, the time series representation calculated in S205 is used as an input feature to train a prediction model that can predict the disk pool usage at the next moment based on the time series representation at the current moment.

[0103] For example, a ridge regression algorithm can be used. Ridge regression is a machine learning algorithm suitable for regression problems. It introduces an L2 regularization term to prevent overfitting, thereby improving the generalization ability of the model. In actual application scenarios, other deep learning algorithms such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or convolutional neural networks (CNNs) can also be considered according to specific application scenarios and data characteristics.

[0104] S207: Evaluate the trained prediction model to obtain an evaluation result.

[0105] After the prediction model is trained, the model can be evaluated to measure its prediction performance and accuracy. In this step, the performance of the disk pool usage prediction model can be evaluated by calculating the RMSE (root mean square error) and MAE (mean absolute error) losses of the prediction results to obtain the evaluation results.

[0106] In an exemplary implementation, the process includes the following steps:

[0107] Step A1: Get an independent test data set that should not overlap with the data set used in the training process. The test data set should contain real disk pool usage data and corresponding timestamps to evaluate the prediction performance of the prediction model on unfamiliar data.

[0108] Step A2: Use the trained prediction model to predict the usage of the hard disk pool at each time point in the test data set to obtain a prediction result.

[0109] Step A3: Calculate RMSE and MAE.

[0110] RMSE is the root mean square of the difference between the predicted value and the true value, which can reflect the degree of dispersion of the prediction error. MAE is the average of the absolute errors between the predicted value and the true value, which can reflect the average size of the prediction error. RMSE and MAE are calculated based on the number of samples in the test data set, the true value of the samples in the test data set, and the predicted value.

[0111] Step A4: Calculate the weighted sum of RMSE and MAE.

[0112] The weights of RMSE and MAE can be freely defined according to business needs and prediction accuracy requirements. For example, if you value the absolute error of the prediction more, you can assign a larger weight to MAE; if you value the discreteness of the prediction error more, you can assign a larger weight to RMSE. Multiply the values ​​of RMSE and MAE by the corresponding weights respectively, and then add the two products to get the final weight sum as the evaluation result of the current prediction model.

[0113] S208: Determine whether the evaluation result meets the first preset requirement.

[0114] The first preset requirement is a pre-set model performance judgment standard, which is used to indirectly judge the training effect of the time series representation extractor and whether to continue training by judging the performance of the prediction model. The first preset requirement can be set according to at least one of the indicators such as accuracy, prediction error threshold, recall rate, mean square error and root mean square error.

[0115] For example, according to the specific requirements of the system, the first threshold is set as an acceptable prediction error threshold as a criterion for judging whether the evaluation result meets the first preset requirement. The first threshold can be an empirical value based on business requirements, or can be calculated based on system performance objectives. The evaluation result obtained in step S207 is compared with the preset first threshold.

[0116] Therefore, the evaluation result in step S207 not only reflects the performance of the prediction model, but also indirectly reflects the effect of the time series representation extractor. If the final evaluation result meets the first preset requirement, that is, the prediction error is less than the first threshold, it means that the features extracted by the time series representation extractor are beneficial to the prediction model and can better reflect the change trend of the hard disk pool usage, then step S209 is executed.

[0117] On the contrary, if the final evaluation result does not meet the first preset requirement, that is, the prediction error is not less than the first threshold, it means that the features extracted by the time series representation extractor are not effective enough, and it is necessary to adjust the self-supervised training strategy and re-pre-train the time series representation extractor, that is, repeat S203 and subsequent steps until the evaluation result of the trained prediction model meets the preset requirements.

[0118] S209: When the evaluation result meets the first preset requirement, stop training the representation extractor, and save the time series representation extractor whose evaluation result meets the first preset requirement.

[0119] By evaluating the performance of the prediction model, it is determined whether to continue training the time series representation extractor. The extractor can efficiently and accurately extract useful features from new time series data to serve tasks such as hard disk capacity planning, performance prediction, or anomaly detection. The time series representation extractor whose evaluation results meet the first preset requirements is deployed on each distributed storage system, so that the time series representation extractor on the distributed storage system deployed by each user is the same.

[0120] In an exemplary implementation, the offline training phase may include the following steps:

[0121] S211. Collect historical monitoring data on usage of hard disk pools of multiple distributed storage systems.

[0122] Collect historical monitoring data of disk pool usage from multiple distributed storage systems. For example, Figure 5 For the offline data set generation process provided in the embodiment of the present application, three different distributed storage systems (i.e. Figure 5 Distributed storage system 1, distributed storage system 2, and distributed storage system 3) in the system, and collect at least 365 days of historical data on disk pool usage from each system. This data may be stored in the system's log files, databases, or dedicated monitoring modules. When collecting this data, it is necessary to ensure the integrity and accuracy of the data for subsequent analysis and modeling.

[0123] S212. Downsample, convert units, fill in missing values, and merge data for the collected historical monitoring data of the hard disk pool.

[0124] After collecting the hard disk pool usage monitoring data in the three distributed storage systems offline, Figure 5 The data is integrated and cleaned to obtain an offline data set.

[0125] In this process, a series of preprocessing operations are required for the data. First, in order to reduce data redundancy and improve computing efficiency, the data is downsampled and the sampling frequency is reduced to one data per day. Secondly, since different systems may use different units to record the usage of the hard disk pool (such as MB, TB, etc.), the units need to be uniformly converted to GB for comparison and analysis. Then, for the missing values ​​that may exist in the data, the forward filling method is used to fill them, that is, the data of the previous day is used to replace the missing values. Finally, the historical monitoring data of the hard disk pool usage of the three distributed storage systems are merged into one data set to form a time series data set of offline hard disk pool usage.

[0126] S213. Based on the offline hard disk pool usage time series dataset, the TS2Vec algorithm is used to train the time series representation extractor.

[0127] The preprocessed offline hard disk pool usage time series dataset is used to train the time series representation extractor. TS2Vec is a self-supervised learning algorithm that can learn meaningful feature representations from time series data. The TS2Vec algorithm is used to train the hard disk pool usage time series data to obtain a model that can extract time series representations.

[0128] S214: Extract the time series representation of the hard disk pool usage based on the trained time series representation extractor.

[0129] Once the time series representation extractor is trained, it can be used to extract time series representations of disk pool usage. These representations are low-dimensional representations of data that can capture key information and patterns in time series data. By extracting time series representations, the original disk pool usage data can be converted into a form that is easier to analyze and model.

[0130] S215 . Based on the extracted time series representation of the hard disk pool usage, a ridge regression algorithm is used to select different ridge regression parameters to train a prediction model, and a first preset number of prediction models are obtained.

[0131] The extracted time series representation of disk pool usage is used to train the ridge regression prediction model. Ridge regression is a linear regression method that introduces regularization terms to prevent overfitting.

[0132] Exemplarily, the first preset number can be 10, that is, in order to find the best model parameters, 10 ridge regression models are trained, and each model uses different ridge regression parameters. By comparing the performance of these models, the best ridge regression parameters and corresponding models can be selected. Among them, the first preset number can be adaptively set and adjusted according to actual needs.

[0133] S216. Evaluate the prediction effect of the ridge regression prediction model based on the weighted sum of RMSE and MAE losses.

[0134] In order to evaluate the performance of the ridge regression prediction model, RMSE (root mean square error) and MAE (mean absolute error) are used as judgment indicators, and the first threshold is set as the maximum threshold of the acceptable RMSE and MAE weight sum as the judgment basis for judging whether the evaluation result meets the first preset requirement. These two indicators measure the degree of deviation and the average absolute deviation between the model prediction value and the true value respectively. The weight between the exemplary RMSE and MAE can be set to 0.5, that is, the preset weight is 0.5, and the weighted sum of RMSE and MAE is calculated based on the preset weight as the final evaluation indicator of the model. Then, 10 ridge regression models are compared, and the minimum weighted value is selected as the evaluation result of the prediction effect. The minimum weighted value is compared with the first threshold, and in response to whether the evaluation result meets the first preset requirement, that is, the minimum weighted value is not less than the first threshold, step S217 is executed. In response to the evaluation result meeting the first preset requirement, that is, the minimum weighted value is less than the first threshold, the hard disk pool usage time series characterization extractor with the minimum weighted value is saved, and subsequently applied to the online training stage and the prediction stage.

[0135] Through this step, the representation extraction capability of the temporal representation extractor can be indirectly evaluated, because the performance of the model directly reflects the quality of the extracted temporal representation.

[0136] S217. Adjust the hyperparameters of TS2Vec and re-execute steps S213 to S216 to train the temporal representation extractor.

[0137] If the performance of the time series representation extractor or ridge regression prediction model obtained by the initial training does not meet the preset requirements, the hyperparameters of TS2Vec, such as learning rate, hidden layer size, etc., can be adjusted according to the evaluation result feedback, and then the time series representation extractor can be retrained. The choice of hyperparameters has an important impact on the performance of the model, so it is necessary to find the best hyperparameter combination through repeated trials. After adjusting the hyperparameters, re-execute steps S213 to S216, retrain the ridge regression prediction model and evaluate its performance. This process may require multiple iterations until the preset requirements are met.

[0138] During the offline training phase, the time series self-supervised contrastive learning algorithm TS2Vec is used to pre-train the disk pool usage time series representation extractor. This is different from the existing supervised algorithm method of directly training the prediction model and then fine-tuning the prediction model. It has the following beneficial effects:

[0139] 1. Traditional supervised learning methods require a large amount of labeled data for training, while TS2Vec can learn with no or a small amount of labeled data. This reduces the dependence on labeled data and makes the algorithm easier to implement and expand in practical applications.

[0140] 2. Compared with traditional statistical time series prediction methods that capture simple linear relationships from historical data, they have low learning freedom and cannot capture nonlinear relationships in complex scenarios. In addition, traditional statistical time series prediction methods require stable data. Therefore, traditional statistical time series prediction methods are suitable for relatively simple scenarios and are not suitable for complex time series patterns.

[0141] TS2Vec can extract more universal time series features because the self-supervised learning method does not rely on specific label data, but learns the intrinsic structure and pattern of the time series data itself to obtain more widely applicable feature representations. This universality enables the feature extractor trained based on TS2Vec to quickly adapt and extract effective features in different application scenarios.

[0142] 3. Compared with the model trained by the traditional statistical time series prediction method, it is applicable to a single time series mode, and a prediction model needs to be established separately for each time series. In the embodiment of the present application, since the features extracted by TS2Vec have higher quality and versatility, the prediction model trained from scratch based on these features can often converge faster and achieve higher prediction accuracy. This is because these features have captured the key information in the time series data, thus providing a better starting point and basis for the prediction model.

[0143] In the online training phase, the prediction model can be trained from scratch for each specific prediction task based on the offline trained representation extractor, which has the following beneficial effects:

[0144] 1. Improve model training efficiency. In the offline stage, there is no pressure for instant prediction, so you can freely choose more complex and computationally intensive algorithms to train the feature extractor. Once the offline training is completed, the resulting time series representation extractor can be regarded as a "model accelerator". In the online training stage, real-time or near real-time data is input into this pre-trained feature extractor, which can be quickly converted into effective features, greatly reducing the complexity and time overhead of online computing. As a result, the online training stage does not need to frequently perform high-cost feature learning steps, but instead focuses on predictions based on extracted features, significantly improving processing speed.

[0145] 2. Enhanced model stability: Since the feature extractor is trained offline, the online stage focuses on updating and training the prediction model, which helps maintain the stability and consistency of the model. Even if the online data changes or fluctuates, since the feature extractor remains unchanged, the prediction model can still make predictions based on stable feature representations, thereby reducing model instability and errors.

[0146] 3. Enhanced flexibility: The offline trained feature extractor can be used as an independent module and combined with different prediction models or algorithms. This allows the online stage to select the appropriate prediction model or algorithm according to actual needs, and flexibly adjust and optimize the performance of the prediction system.

[0147] 4. Enhanced scalability: As time goes by and the amount of data increases, the offline trained feature extractor can be regularly updated to better adapt to new data and new situations.

[0148] The online training phase is described below. Figure 6The schematic diagram of the online training phase process provided by the embodiment of the present application is executed by the training module of the distributed storage system. The tasks of the training module can be deployed on any computing device with sufficient resources in the distributed storage system. The computing device can be dynamically adjusted according to the task requirements and its own resource status to achieve load balancing. Among them, the representation extractor can be expressed as a time series representation extractor, and the online data can be expressed as an online data set, such as Figure 6 As shown, the online training phase includes:

[0149] S301, obtaining time series data of hard disk pool usage.

[0150] During the normal operation of the distributed storage system, at least one computing device in the distributed storage system can execute the process of the online training phase and periodically collect the time series data of the disk pool usage. Exemplarily, the subsequent process of the online training phase can be executed by the training module deployed on the computing device. The data collected in the online training phase reflects the usage of the disk pool at different time points, providing a basis for subsequent prediction analysis.

[0151] S302: Determine whether the system operation time meets the preset time.

[0152] Before making a prediction, the training module of the distributed storage system needs to determine whether the system's operating time meets the preset requirements. If it does not meet the preset time, no prediction is made, and the system continues to run normally until the preset time is met. If it does, step S33 is executed.

[0153] In order to ensure the integrity of the data and the stability of the system, it is necessary to determine whether the system running time meets the preset requirements. If the system running time is insufficient, it may mean that the data accumulation is not sufficient and it is not suitable for predictive analysis at this time. Therefore, the system will continue to wait until the running time meets the preset conditions.

[0154] S303: Pull the full amount of online hard disk pool usage time series data.

[0155] When the system operation time meets the requirements, the training module pulls the full amount of online disk pool usage time series data. This data is updated in real time and can reflect the latest usage of the disk pool. By obtaining the full amount of data, the accuracy and comprehensiveness of subsequent analysis can be ensured.

[0156] S304: Preprocess the hard disk pool usage time series data to obtain an online data set.

[0157] Preprocessing helps remove noise and outliers from data and improves the quality and usability of data.

[0158] Exemplarily, preprocessing may include downsampling, unit conversion, and missing value filling to obtain one or more of the online hard disk pool usage time series data.

[0159] S305 . Determine, based on the time series characterizer, the time series characterization corresponding to the hard disk pool usage time series data in the online data set.

[0160] The training module extracts the time series representation of the online hard disk pool usage based on the hard disk pool usage time series representation extractor obtained in the offline training phase. Specifically, the time series representation at each moment is calculated.

[0161] S306 , training a prediction model based on the time series representation and a second preset number of linear regression prediction algorithms.

[0162] The prediction model is trained based on the time series representation extracted in the above steps and the second preset number of linear regression prediction algorithms. Exemplarily, the second preset number can be 4, that is, 4 linear regression prediction algorithms are used in parallel, which can include ordinary least squares, ridge regression, lasso regression, and elastic network regression. Each linear regression prediction algorithm executes steps S306 to S307 in parallel. Among them, the second preset number can be adaptively set and adjusted according to actual needs.

[0163] S307. Optimize the hyperparameters of the prediction model based on the optimization algorithm.

[0164] That is, the prediction models corresponding to the linear regression prediction algorithms are optimized according to the optimization algorithm, and steps S306 and S307 are continuously executed until the number of optimizations reaches a third preset number, where the third preset number is a preset number of optimizations that each prediction model needs to complete.

[0165] Exemplarily, the RMSE and MAE losses can be calculated for the prediction results of the time series prediction model, and the weighted sum of the two can be taken as the evaluation result, and the weight can be adaptively defined according to actual needs. The optimization algorithm can use the fruit fly optimization algorithm, which optimizes the hyperparameters of the linear regression prediction model based on the evaluation results. Each linear regression prediction model completes the third preset number of optimizations and trainings to obtain the prediction model corresponding to each linear regression prediction algorithm.

[0166] S308: Save the prediction model that meets the second preset requirement.

[0167] The second preset requirement is a pre-set performance judgment standard of the prediction model, which is used to judge whether the performance of the prediction model obtained by the current training needs to be further trained. When the prediction model obtained by the training meets the second preset requirement, the prediction model can be used to predict the prediction data. The second preset requirement can be set according to at least one of the indicators such as accuracy, mean square error, and mean absolute error.

[0168] Exemplarily, the prediction model with the smallest prediction error can be screened by judging whether the prediction model meets the second preset requirement. For example, after training four linear regression prediction models, the training module can select the prediction model with the smallest prediction error among the four linear regression prediction models according to the prediction error, and use it to predict the hard disk pool usage data in the future period in the online prediction stage.

[0169] In an exemplary implementation, the online training phase may include the following steps S311-S316:

[0170] S311, periodically executing the online training phase to obtain a hard disk pool usage time series prediction model.

[0171] To ensure the stability and accuracy of the prediction model, a periodic online training process is set. After the distributed storage system has been running normally for at least 30 days, the training module in the distributed storage system starts to execute this process. If the system has been running for less than 30 days, model training and prediction will not be performed, and the disk pool usage monitoring curve on the system page will not display the prediction data. The training cycle is set to 7 days, that is, the complete online training process is executed every 7 days, and the current optimal prediction model is updated. This periodic training helps the model adapt to the dynamic changes of the system and maintain its prediction performance.

[0172] S312. When the distributed storage system meets the condition of normal operation for at least 30 days, the training module starts to pull the full amount of hard disk pool usage time series data. A series of preprocessing operations are performed on this data, including data downsampling, unit conversion, and missing value filling. Downsampling is to reduce data redundancy and improve computing efficiency. The sampling frequency is set to one data per day. The unit is uniformly converted to GB for unified analysis and comparison. For missing values, the forward filling method is adopted, that is, the data of the previous day is used to fill the missing values.

[0173] S313. Based on the time series representation extractor trained in the offline stage, the training module rolls over the time series representation of the online hard disk pool usage. The rolling window is 7 days, that is, one feature is calculated for every 7 days of hard disk pool usage data as the time series representation of the last day.

[0174] S314: Based on the time series representation of the online hard disk pool usage, train a time series prediction model. Train four linear regression time series prediction models, including ordinary least squares, ridge regression, lasso regression, and elastic network regression. Specifically, based on the time series features calculated from the past 7 days of data in S313, predict the hard disk pool usage for the next day.

[0175] Each linear regression prediction algorithm executes step S314 to step S315 in parallel.

[0176] S315. Optimize the hyperparameters of the prediction model based on the fruit fly optimization algorithm.

[0177] Specifically, the RMSE and MAE losses are calculated for the prediction results of the time series prediction model, and the weighted sum of the two is taken as the evaluation result, and the weights of RMSE and MAE are both 0.5.

[0178] The fruit fly optimization algorithm optimizes the hyperparameters of the linear regression prediction model based on the evaluation results, and saves the optimal prediction model of each linear regression prediction algorithm.

[0179] Each linear regression prediction algorithm is trained 10 times in each iteration of hyperparameter optimization, and the optimal time series prediction model is taken.

[0180] S316. Among the four linear regression prediction models, the globally optimal hard disk pool usage time series prediction model is saved, and is used to predict the hard disk pool usage data in the future in the online prediction stage.

[0181] The above online training phase has the following beneficial effects:

[0182] 1. In the online phase, the prediction model is trained based on the actual data of each user, ensuring that the model can capture the unique storage usage patterns of different users. This customized approach can better reflect the real needs and behaviors of users than general models, and improves the pertinence and accuracy of predictions.

[0183] 2. By training a variety of different linear regression models and selecting the optimal prediction model, the diversity of model adaptability is increased, which can more flexibly match the user's unique data distribution and trends, avoid the limitations that may exist in a single model, and improve the universality and accuracy of the prediction.

[0184] 3. Training multiple models in parallel significantly reduces the total training time, fully utilizes the concurrent capabilities of computing resources, and speeds up the model deployment and update cycle, which is crucial for online systems that need to respond quickly to user needs.

[0185] 4. Use the fruit fly optimization algorithm to adjust the model hyperparameters online, and dynamically find the best configuration of the model. Compared with fixed hyperparameters or static optimization methods such as grid search, the fruit fly algorithm can reach a better local optimal solution in a shorter time, adapt to changes in user storage behavior, and improve the adaptability and prediction efficiency of the model.

[0186] 5. Regularly training and updating the prediction model ensures the timeliness of the model and can promptly reflect the latest changes in user storage usage patterns. This continuous learning mechanism is an effective strategy for dealing with data dynamics and avoids model obsolescence and failure due to changes in data distribution.

[0187] 6. Periodically train and update the disk pool usage prediction model. Compared with using the prediction model after training, this method can ensure that the prediction model captures the user's latest storage usage pattern, because the user's storage usage pattern may change over time.

[0188] Figure 7 The schematic diagram of the online prediction phase process provided by the embodiment of the present application is executed by the prediction module of the distributed storage system. The tasks of the prediction module can be deployed on any computing device with sufficient resources in the distributed storage system. The computing device can be dynamically adjusted according to the task requirements and its own resource status to achieve load balancing. Among them, the representation extractor can be expressed as a time series representation extractor, and the prediction data can be expressed as a prediction data set. Figure 7 As shown in the figure, the online prediction process includes:

[0189] S401, obtaining hard disk pool usage time series data.

[0190] During the operation of the distributed storage system, the prediction module in the distributed storage system continuously and periodically collects the usage data of the hard disk pool. The frequency of collecting this data may be determined according to the specific needs of the system, such as collecting once every hour, every day, or every week. The data collection method may be achieved through the monitoring tools within the system or a dedicated logging system. The collected data will be recorded and stored for subsequent analysis and prediction.

[0191] In a possible implementation, during the operation of the distributed storage system, the prediction module in the distributed storage system continuously and periodically collects the usage data of the hard disk pool. In response to the prediction trigger instruction and / or the storage amount of the hard disk pool usage time series data meets the prediction start condition, the hard disk pool usage time series data used for predicting the hard disk pool usage is obtained.

[0192] The prediction trigger instruction is an external signal used to start the prediction process, which can be an instruction issued by the user. In actual application scenarios, users or system administrators can manually trigger the prediction process based on business needs. In this case, the user can click a button or select a menu item on the front-end page of the distributed storage system to send a prediction trigger instruction for instructing the prediction process to be started to the prediction module in the distributed storage system.

[0193] In a possible implementation, the prediction process can be triggered by only satisfying the prediction trigger instruction condition. Then, when the prediction trigger instruction is obtained, S403 and subsequent steps are executed.

[0194] In a possible implementation, the prediction process can be triggered only when the prediction trigger instruction and the storage volume of the hard disk pool usage time series data meet the prediction start condition. When the prediction trigger instruction is obtained, S402 and subsequent steps are executed.

[0195] In a possible implementation, the prediction process can be triggered only if the storage volume of the hard disk pool usage time series data meets the prediction start condition. Then, it is not necessary to determine whether a prediction trigger instruction is received, and S402 and subsequent steps are executed.

[0196] S402: Determine whether the storage capacity of the hard disk pool usage time series data meets the prediction start condition.

[0197] Among them, the prediction start condition can be set according to the storage capacity threshold of the hard disk pool usage time series data. When the storage capacity of the hard disk pool usage time series data is not less than the storage capacity threshold, that is, the storage capacity of the hard disk pool usage time series data meets the prediction start condition, and S403 and subsequent steps are executed; when the storage capacity of the hard disk pool usage time series data is less than the storage capacity threshold, that is, the storage capacity of the hard disk pool usage time series data does not meet the prediction start condition, no prediction is performed, and the distributed storage system is waited to continue to obtain the hard disk pool usage data during normal operation until the storage capacity of the hard disk pool usage time series data meets the prediction start condition.

[0198] In a possible implementation, the running time of the distributed storage system can be used to determine whether the storage amount of the disk pool usage time series data meets the prediction start condition. Before making a prediction, the prediction module of the distributed storage system needs to determine whether the running time of the system meets the preset time.

[0199] The preset duration can be set based on system stability and data accumulation. If the system running time is insufficient, the system can determine that the storage volume of the current disk pool usage time series data is insufficient to support effective prediction, so it will choose to continue waiting until the system running time reaches the preset duration, that is, the storage volume of the disk pool usage time series data meets the preset startup conditions. This process ensures that the prediction model is trained and applied with sufficient data support.

[0200] The prediction module determines whether the storage capacity of the disk pool usage time series data meets the prediction start condition through the running time of the distributed storage system. If not, no prediction is performed and the system continues to run normally until the preset time is met. If so, step S403 is executed.

[0201] S403: Pull required online hard disk pool usage time series data.

[0202] When the system operation time meets the requirements, the prediction module pulls the online disk pool usage time series data. This data is real-time and recent, and can reflect the current disk pool usage. Compared with full data, pulling online data can greatly reduce the complexity of data processing and analysis and improve the efficiency of prediction.

[0203] S404: Preprocess the hard disk pool usage time series data to obtain a prediction data set.

[0204] The prediction module preprocesses the disk pool usage time series data, which may include downsampling, unit conversion, and missing value filling, to obtain a prediction data set including the online disk pool usage time series data. For example, the number of data points can be reduced through downsampling technology, which helps to reduce the computational cost of subsequent analysis. Secondly, unit conversion is performed to ensure that all data uses a unified measurement unit for comparison and analysis. Finally, the missing value problem is handled, and the gaps in the data are filled through interpolation or other methods to ensure the integrity and accuracy of the data.

[0205] S405 . Determine the time series representation corresponding to the prediction data set based on the time series representation extractor.

[0206] In the offline training phase, a disk pool usage time series representation extractor has been trained. This extractor can extract useful features or representations from the original time series data. These representations can capture the inherent laws and patterns of the data and provide key information for subsequent predictions. In the online phase, the prediction module uses this extractor to process the prediction data set obtained in the above steps to obtain its corresponding time series representation.

[0207] S406: Based on the prediction model and time series representation, a rolling prediction is performed on the disk pool usage data in the future.

[0208] Based on the disk pool usage time series prediction model obtained during the online training phase and the extracted time series representation, the prediction module begins to make rolling predictions for the disk pool usage in the future. Rolling prediction means that the prediction is continuously updated over time. The system can re-predict periodically (such as every hour or every day) to reflect the latest disk pool usage. This approach ensures the timeliness and accuracy of the prediction results.

[0209] S407: Save the prediction result to the cache module.

[0210] The prediction results can be saved to the cache module for subsequent use and analysis. The cache module refers to a system dedicated to storing temporary data. The stored data is usually frequently accessed or has high computational costs. It can quickly respond to requests from other components and provide the required prediction data. The main purpose of the cache module is to improve the response speed and performance of the system by reducing the number of direct accesses to the original data source (such as a database or remote service).

[0211] In an exemplary implementation, the online prediction stage may include the following steps S411-S412:

[0212] S411. The distributed storage system periodically starts prediction to obtain prediction results.

[0213] After the distributed storage system has been running normally for 30 days, the system will automatically start the disk pool usage forecasting function. The forecasting function is executed 4 times a day, that is, once every 6 hours. Such periodic forecasting can ensure that the disk pool usage data is kept up to date and meet the needs of real-time monitoring and management.

[0214] The result of each prediction will be saved in the cache module, and the cache duration is set to 1 day. In this way, even when the prediction model cannot access real-time data immediately, the most recent prediction result can be provided through the cache.

[0215] S412: Pull the online hard disk pool usage time series data for the last 7 days to predict the hard disk pool usage data for the next day, and perform rolling prediction.

[0216] The prediction model pulls the online disk pool usage time series data of the last 7 days as input to predict the disk pool usage data for the next day. This rolling window-based approach ensures that the model always makes predictions based on the latest data. The prediction model can perform rolling predictions, that is, after each prediction is completed, the latest prediction results will be added to the data set and the oldest data will be removed to maintain the timeliness and accuracy of the data set.

[0217] The predicted future duration will be adjusted according to the historical duration of the distributed storage system operation. That is, after 30 days of normal system operation, the disk pool usage for the next 15 days can be predicted (half of the historical system operation duration). As the system operation time increases, the predicted duration will also increase accordingly. The disk pool usage for the next 90 days can be predicted at most after the system has been running for 180 days.

[0218] In an exemplary implementation, a distributed storage system may set up a front-end page, such as a central control plane or a front-end portal, which may be a web interface or an API service. Users do not directly interact with each computing device on the back end, but may interact with the system through the front-end page and submit requests, such as data queries, model training tasks, or prediction requests.

[0219] In the online prediction stage, the process of the distributed storage system front-end requesting prediction data from the back-end is as follows: Figure 8 As shown, Figure 8 A schematic diagram of a request prediction data flow provided in an embodiment of the present application, the flow includes S501-S502:

[0220] S501. The distributed storage system front end sends a query request to the distributed storage system back end.

[0221] The query request is a request for querying the disk pool usage forecast data. The user or administrator submits a query request through the front-end page of the distributed storage system. The request is used to query the disk pool usage forecast data, indicating that the user currently needs to know the disk pool usage forecast data for a period of time in the future. This request is sent to the system backend for processing.

[0222] S502: The backend of the distributed storage system determines whether the normal operation time of the distributed storage system meets the preset time.

[0223] After receiving the request, the backend first determines whether the current system has been running stably for the preset minimum time. This step is to ensure the reliability of the prediction results, because the disk usage pattern of a newly deployed system or a system that has just undergone major changes may not be stable, which will affect the accuracy of the prediction.

[0224] If the system has been running stably for a long enough time, step S503 is executed.

[0225] S503: The back end of the distributed storage system queries the cache module for disk pool usage prediction data, and determines whether the cache module has the prediction data.

[0226] If there is predicted data corresponding to the query request in the cache module, step S504 is executed to return the data corresponding to the query request in the cache module to the front end to complete the request response. The whole process is efficient and fast.

[0227] If no prediction data is found in the cache, the backend will trigger the online prediction process. The pre-trained prediction model is called, and the current disk pool usage status and other related parameters are input to predict the usage in the future. After the prediction is completed, not only the results are returned to the front-end user, but also the prediction data is saved in the cache module to provide a basis for a quick response for the next same or similar request.

[0228] S504: The distributed storage system backend returns data corresponding to the query request in the cache module to the distributed storage system frontend.

[0229] That is, the data corresponding to the query request in the cache module is returned to the front end to complete the prediction request response.

[0230] In a possible implementation, the distributed storage system front end draws a prediction result curve based on the data returned by the distributed storage system back end. Exemplarily, the prediction result curve is drawn using an appropriate chart library based on the type and format of the data and displayed on a display interface of the distributed storage front end.

[0231] The process in which the front end of the distributed storage system page requests prediction data from the back end has the following beneficial effects:

[0232] 1. Reduce the computational burden: By introducing the use of the cache module, the frequent occupation of computing resources can be significantly reduced, especially when prediction requests are frequent.

[0233] 2. Improve response speed: For most queries, especially those repeated in a short period of time, you can get the results directly from the cache, with almost instant response.

[0234] 3. Flexibility and real-time performance: The online prediction mechanism ensures that the latest prediction data can be obtained in time even when it is the first query or the cache expires, thus ensuring the timeliness of the prediction.

[0235] 4. Resource optimization: Combine runtime checks with caching strategies to effectively utilize computing resources and avoid unnecessary calculations, especially when the system is just started or in the early stages of data accumulation.

[0236] The "first" and "second" (if any) in the names mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0237] The above are some specific implementation methods of the hard disk pool usage prediction method provided in the embodiment of the present application. Based on this, the embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer implements the hard disk pool usage prediction method provided in the embodiment of the present application.

[0238] An embodiment of the present application also provides a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to implement the hard disk pool usage prediction method provided in the embodiment of the present application.

[0239] The computer storage medium stores codes, and when the codes are executed, the device executing the codes implements the hard disk pool usage prediction method described in any embodiment of the present application.

[0240] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0241] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0242] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for predicting hard disk pool usage, characterized in that: The method comprises a training phase and a prediction phase: The training phase includes: Acquire offline data, and train a representation extractor based on the offline data to obtain a trained representation extractor; Acquire online data, and use the trained representation extractor to extract the online data to obtain time series features, and train a prediction model based on the time series features to obtain a trained prediction model; The prediction stage includes: acquiring prediction data, the prediction data including hard disk pool usage time series data for predicting hard disk pool usage; extracting the time series representation corresponding to the prediction data based on the trained representation extractor; inputting the time series representation corresponding to the prediction data into the trained prediction model to obtain a prediction result, and the prediction result is used to represent the size of the hard disk pool usage.

2. The method according to claim 1, characterized in that The step of training the representation extractor based on the offline data to obtain a trained representation extractor includes: Training a representation extractor using the offline data; Extracting the time series representation of the offline data based on the time series representation extractor; Training a prediction model using the time series representation of the offline data; Evaluate the trained prediction model to obtain evaluation results; When the evaluation result meets the first preset requirement, stop training the representation extractor; when the evaluation result does not meet the first preset requirement, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the evaluation result meets the first preset requirement.

3. The method according to claim 2, characterized in that The method of using the offline data to train a representation extractor comprises: A representation extractor is trained based on the time series self-supervised learning algorithm TS2Vec and the offline data; When the evaluation result does not meet the first preset requirement, updating the model parameters of the representation extractor, and continuing to train the time series representation extractor using the offline data until the evaluation result meets the first preset requirement, includes: When the evaluation result does not meet the first preset requirement, the hyperparameters of the TS2Vec algorithm are updated based on the evaluation result, and the training representation extractor based on the TS2Vec algorithm and the offline data continues until the evaluation result meets the first preset requirement.

4. The method according to claim 2, characterized in that: The evaluation result includes a prediction error, and the evaluation of the trained prediction model to obtain the evaluation result includes: Based on the prediction model, the usage of the hard disk pool at each time point in the evaluation data set is predicted to obtain the prediction result; Calculating a prediction error of the prediction result; When the evaluation result meets the first preset requirement, the training of the representation extractor is stopped; when the evaluation result does not meet the first preset requirement, the model parameters of the representation extractor are updated, and the training of the time series representation extractor using the offline data continues until the evaluation result meets the first preset requirement, including: When the prediction error is less than a first threshold, stop training the representation extractor; when the prediction error is not less than the first threshold, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the prediction error is less than the first threshold.

5. The method according to claim 4, characterized in that The prediction error includes a weighted sum of a root mean square error and a mean absolute error of the prediction result, and the calculation of the prediction error of the prediction result includes: Calculating the root mean square error and mean absolute error of the prediction results; Based on the preset weights, a weighted sum of the root mean square error and the mean absolute error of the prediction result is calculated.

6. The method according to claim 2, characterized in that The method of training a prediction model using the time series representation of the offline data includes: Based on the time series representation of the offline data, using a ridge regression algorithm to select different ridge regression parameters to train a prediction model, to obtain a first preset number of prediction models; The step of evaluating the trained prediction model to obtain an evaluation result includes: Calculating the prediction error of each prediction model in the first preset number of prediction models; Obtain the minimum prediction error value in the calculation results as the evaluation result; When the evaluation result meets the first preset requirement, the training of the representation extractor is stopped; when the evaluation result does not meet the first preset requirement, the model parameters of the representation extractor are updated, and the training of the time series representation extractor using the offline data continues until the evaluation result meets the first preset requirement, including: When the minimum prediction error value in the calculated results is less than a first threshold, stop training the representation extractor; when the minimum prediction error value in the calculated results is not less than the first threshold, update the model parameters of the representation extractor, and continue to train the time series representation extractor using the offline data until the prediction error is less than the first threshold.

7. The method according to claim 1, characterized in that The step of training the prediction model based on the time series features to obtain a trained prediction model includes: A prediction model is obtained based on the time series representation training of the online data; The trained prediction model is evaluated to determine a prediction model that meets the second preset requirement, and the prediction model that meets the second preset requirement is used to predict the prediction data.

8. The method according to claim 7, characterized in that The prediction model is obtained by training the time series representation of the online data, including: Using the online data and a second preset number of linear regression prediction algorithms to train a prediction model, a first preset number of prediction models corresponding to each linear regression prediction algorithm is obtained; Optimizing the prediction models corresponding to the linear regression prediction algorithms based on the optimization algorithm, and continuing to train the prediction models using the online data and the first preset number of linear regression prediction algorithms until the number of optimizations reaches a third preset number; The step of evaluating the trained prediction model to determine a prediction model that meets the second preset requirement includes: The prediction errors of the prediction models corresponding to the various linear regression prediction algorithms are calculated, and the prediction model with the smallest prediction error is determined. The prediction model with the smallest prediction error is used to predict the prediction data.

9. The method according to claim 1, characterized in that: The obtaining of prediction data comprises: In response to the prediction trigger instruction, and / or the storage amount of the hard disk pool usage time series data meets the prediction start condition, obtaining the hard disk pool usage time series data for predicting the hard disk pool usage; The hard disk pool usage time series data used for predicting the hard disk pool usage is preprocessed to obtain the prediction data.

10. A server, characterized in that: It includes a processor and a memory, the processor and the memory are connected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the hard disk pool usage prediction method as described in any one of claims 1 to 9.