Storage capacity water level risk management method and device and electronic equipment

Abstract

The invention provides a storage capacity water level risk management method and device and electronic equipment, and the method comprises the steps: obtaining the capacity condition, flow condition and other storage parameter information of a to-be-processed data cluster; the storage parameter information is input into an LSTM neural network model obtained through training in advance based on historical storage parameter information and historical storage capacity water level real conditions, the time sequence model carries out data processing based on the input storage parameter information, a storage water level prediction result of the data cluster is output, and then a storage water level prediction result of the data cluster is obtained based on the storage water level prediction result. And determining a target capacity management scheme and executing the target capacity management scheme. By selecting the embodiment of the invention, the capacity water level risk is predicted through the time sequence neural network, and an automatic capacity water level early warning and capacity management scheme is provided, so that the hysteresis and errors existing in manual decision making are reduced, and the capacity management decision making efficiency and response speed are improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a storage capacity water level risk management method, device and electronic equipment. Background Art

[0002] In the field of data storage technology, storage capacity level refers to the storage capacity usage of a storage system or storage device. Storage capacity level risk management is key to ensuring the normal operation of storage systems or storage devices. For example, if the storage capacity level of a storage system gradually increases, it will affect the normal response of data services using the storage system. In this case, if the storage capacity level of the storage system rises to a certain level, the storage capacity of the storage system needs to be expanded to ensure that the storage system can normally respond to data requests from the data services deployed in the storage system.

[0003] However, common capacity and water level risk management solutions currently in use often rely on administrators to manually calculate and forecast water levels, and then manually decide on the timing and quantity of storage capacity expansion and contraction. This often lacks in-depth analysis and prediction of complex data relationships, leading to lags in the adopted capacity expansion and contraction solutions and waste of resources. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a storage capacity water level risk management method, device and electronic device, which can automatically analyze the storage capacity water level and provide accurate storage capacity water level prediction results, so as to accurately expand or shrink the storage capacity.

[0005] In a first aspect, an embodiment of the present application provides a storage capacity water level risk management method, the method comprising:

[0006] Acquire various storage parameter information of the data cluster to be processed, wherein the storage parameter information includes: the capacity of the data cluster and the flow of the data cluster;

[0007] Inputting each of the storage parameter information into a preset time series model, the time series model performs data processing based on the input storage parameter information, and outputs a storage water level prediction result for the data cluster; wherein the preset time series model is an LSTM neural network model pre-trained using the historical storage parameter information of the data cluster and the actual historical storage capacity water level;

[0008] Based on the storage water level prediction result, a target capacity management plan is determined and executed, wherein the target capacity management plan includes: expansion or reduction, the capacity required for the expansion or the capacity required for the reduction.

[0009] In combination with the first aspect, in a second possible embodiment, the preset time series model is pre-trained through the following steps:

[0010] Obtain historical storage parameter information and historical storage capacity level information of the data cluster;

[0011] Constructing a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level real situation;

[0012] Inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold;

[0013] The LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold is determined as the preset time series model.

[0014] In combination with the first aspect, in a third possible embodiment, the capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the new quota capacity applied for by the user, and the optimized quota capacity applied for by the user; the traffic situation of the data cluster includes but is not limited to: the read and write traffic situation of the user storage container and the request volume of the user storage container.

[0015] In combination with the first aspect, in a fourth possible embodiment, the method further includes:

[0016] Performing data cleaning on the capacity of the data cluster and the flow of the data cluster, and constructing a storage parameter information feature matrix;

[0017] Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function;

[0018] Based on the storage water level prediction probability, a storage water level prediction result of the data cluster is determined.

[0019] In combination with the first aspect, in a fifth possible embodiment, the method is applied to a data cluster of object storage, and the method further includes:

[0020] Respond to data flow requests based on the address of the expanded or reduced storage container.

[0021] In a second aspect, an embodiment of the present application provides a storage capacity water level risk management device, wherein the device includes:

[0022] An input module is used to obtain various storage parameter information of the data cluster to be processed, wherein the storage parameter information includes: the capacity of the data cluster and the flow of the data cluster;

[0023] A prediction module is configured to input each of the storage parameter information into a preset time series model, and the time series model performs data processing based on the input storage parameter information to output a storage water level prediction result for the data cluster; wherein the preset time series model is an LSTM neural network model pre-trained using the historical storage parameter information of the data cluster and the actual historical storage capacity and water level;

[0024] A management module is used to determine a target capacity management plan based on the storage water level prediction result and execute the target capacity management plan, wherein the target capacity management plan includes: expansion or reduction, the capacity required for the expansion or the capacity required for the reduction.

[0025] In conjunction with the second aspect, in a second possible embodiment, the apparatus further includes: a model training module, the model training module being configured to:

[0026] Obtain historical storage parameter information and historical storage capacity level information of the data cluster;

[0027] Constructing a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level real situation;

[0028] Inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold;

[0029] The LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold is determined as the preset time series model.

[0030] In combination with the second aspect, in a third possible embodiment, the capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the new quota capacity applied for by the user, and the optimized quota capacity applied for by the user; the traffic situation of the data cluster includes but is not limited to: the read and write traffic situation of the user storage container and the request volume of the user storage container.

[0031] In conjunction with the second aspect, in a fourth possible embodiment, the prediction module is further configured to:

[0032] Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function;

[0033] Based on the storage water level prediction probability, a storage water level prediction result of the data cluster is determined.

[0034] In conjunction with the second aspect, in a fifth possible embodiment, the apparatus is applied to a data cluster of object storage, and the apparatus further includes:

[0035] The request migration module is used to respond to data flow requests based on the address of the storage container after expansion or reduction.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes: a processor; and a memory for storing a program; wherein the program includes instructions, which, when executed by the processor, enable the processor to execute the storage capacity water level risk management method described in the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the storage capacity water level risk management method described in the first aspect.

[0038] Beneficial effects of this application:

[0039] The present application provides a storage capacity water level risk management method, device and electronic device, wherein the method obtains various storage parameter information such as the capacity and flow conditions of the data cluster to be processed, and inputs the storage parameter information into an LSTM neural network model that has been pre-trained based on historical storage parameter information and the actual conditions of historical storage capacity water levels. The time series model processes data based on the input storage parameter information, outputs the storage water level prediction result of the data cluster, and then determines the target capacity management plan based on the storage water level prediction result and executes the target capacity management plan. The embodiment of the present application is selected to predict capacity water level risks through a time series neural network, and provide automated capacity water level warnings and capacity management plans, which helps to reduce the lag and errors in manual decision-making and improve the efficiency and response speed of capacity management decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A flow chart of a storage capacity water level risk management method provided in an embodiment of the present application is shown;

[0042] Figure 2 A schematic diagram of a flow chart of a method for training a preset time series model provided in an embodiment of the present application is shown;

[0043] Figure 3 Another flow chart of the storage capacity water level risk management method provided in an embodiment of the present application is shown;

[0044] Figure 4 A schematic diagram of a logical architecture of a storage capacity and water level prediction and management device provided in an embodiment of the present application is shown;

[0045] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present application is shown. DETAILED DESCRIPTION

[0046] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0047] It should be understood that the various steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0048] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0049] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0050] In order to solve the problems of lag and resource waste in the existing technology of using manual capacity water level prediction and manual decision-making for expansion and contraction plans, this application provides a storage capacity water level risk management method, device and electronic equipment. In the first aspect, this application provides a storage capacity water level risk management method, which is applied to any electronic device with storage capacity water level risk management function, including but not limited to personal mobile terminals, computers or servers, etc. Figure 1 As shown, the method includes the following steps:

[0051] S11. Obtain various storage parameter information of the data cluster to be processed.

[0052] The storage parameter information includes: the capacity of the data cluster and the flow of the data cluster.

[0053] S12: Input each of the storage parameter information into a preset time series model, and the time series model performs data processing based on the input storage parameter information to output a storage water level prediction result of the data cluster.

[0054] Among them, the preset time series model is an LSTM neural network model that is pre-trained using historical storage parameter information of the data cluster and the actual historical storage capacity water level situation.

[0055] S13. Determine a target capacity management plan based on the storage water level prediction result, and execute the target capacity management plan.

[0056] The target capacity management scheme includes: capacity expansion or capacity reduction, the capacity quantity required for the capacity expansion, or the capacity quantity required for the capacity reduction.

[0057] The method obtains various storage parameter information such as the capacity and flow conditions of the data cluster to be processed, and inputs the storage parameter information into an LSTM neural network model that has been pre-trained based on historical storage parameter information and the actual conditions of historical storage capacity water levels. The time series model processes the data based on the input storage parameter information, outputs the storage water level prediction result of the data cluster, and then determines the target capacity management plan based on the storage water level prediction result and executes the target capacity management plan. The embodiment of the present application is selected to predict capacity water level risks through a time series neural network, and to provide automated capacity water level warnings and capacity management plans, which helps to reduce the lag and errors in manual decision-making and improve the efficiency and response speed of capacity management decisions.

[0058] The following will describe the above steps S11 to S13 in detail with reference to specific implementation examples:

[0059] In the embodiment of the present application, the data cluster to be processed refers to a data cluster that needs to perform capacity water level prediction. As an implementation method, the storage capacity water level risk management method provided in the embodiment of the present application can be applied to the management device of the data cluster, and can also be applied to the management device of the data system composed of multiple data clusters. Among them, the various storage parameter information in the data cluster refers to the various parameters associated with the description of the capacity water level changes of the data cluster, which can be divided into two categories: capacity situation and traffic situation. The capacity situation refers to the macro and micro capacity usage of the current data cluster, and the traffic situation refers to the specific situation of the access traffic of the current data cluster. Among them, the macro capacity usage refers to the capacity situation of the entire data cluster, and the micro capacity usage refers to the capacity situation of each storage unit in the data cluster. As an implementation method, the data cluster is an object storage data cluster, and the storage unit in the data cluster is specifically a container Bucket for storing data objects.

[0060] Based on this, as an implementation method, the capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the new quota capacity applied for by the user, and the optimized quota capacity applied for by the user.

[0061] Among them, allocated capacity refers to the capacity already allocated for object storage, actual usage of user storage containers refers to the capacity of the bucket actually used by the user, and actual usage of management storage containers refers to the capacity of the bucket actually used by the administrator. User storage container quota capacity refers to the capacity of the storage container bucket allocated to the user in response to the user's storage request, user-applied new quota capacity refers to the capacity of the new user storage container bucket requested by the user, and user-applied optimized quota capacity refers to the capacity of the user storage container bucket requested to be reduced or optimized.

[0062] Among them, as an implementation method, in the process of executing step S11, the capacity of the data cluster can be obtained through the management console, API (Application Programming Interface) or SDK (Software Development Kit), command line tool or monitoring and analysis function module provided by the cloud vendor. As an implementation method, the various capacity conditions of the data cluster used are obtained through the API interface provided by the cloud vendor through crawler technology. As an example, the list of each Bucket in the object storage can be obtained through the API interface, and then the Bucket storage capacity and the change of the Bucket storage capacity can be obtained from the Bucket list. Specifically, for BOS object storage, the GET / storageStat interface can be used to directly return the total storage volume of the Bucket, the number of objects, and the proportion of each storage type.

[0063] The data cluster's traffic includes, but is not limited to, the read and write traffic of the user storage container and the request volume of the user storage container. The read and write traffic of the user storage container refers to the traffic volume of read and write requests to the user storage bucket. The request volume of the user storage container refers to the number of various requests to the user storage bucket.

[0064] Among them, in the process of executing step S11, similar to the means of obtaining capacity conditions, traffic conditions can also be obtained through the management console, API (Application Programming Interface) or SDK (Software Development Kit), command line tool or monitoring and analysis function module provided by the cloud vendor. Real-time inflow and outflow traffic can be obtained, and real-time request volume can also be obtained, and the request volume includes write PUT requests, read GET requests, and delete DELETE requests. As an implementation method, the API provided by the cloud vendor can be used to use crawler technology to obtain the traffic and request conditions corresponding to each Bucket in the object storage, and then obtain the traffic and request conditions of the total Bucket, a single Bucket or a combined Bucket.

[0065] As an implementation method, the capacity and traffic conditions obtained each time can be stored separately to store the historical storage parameter information of the data cluster, so as to perform model training based on the historical storage parameter information and explore more intelligent storage management possibilities.

[0066] Based on this, when executing step S12, the various storage parameter information obtained in step S11 is used as input to a preset time series model. The preset time series model processes the input storage parameter information and outputs the storage water level prediction result of the data cluster. Among them, the preset time series model can be regarded as a black box for data processing, which is a pre-trained neural network model with storage water level prediction capabilities. Among them, the time series model can also be called a time series neural network model, which is a neural network model specially designed for processing time series data. Time series data usually has time correlation, in which the current state of the data is related to the past state. This type of model is widely used in various time series data prediction, analysis and modeling, such as stock prediction, weather broadcasting, natural language processing, etc. Common time series neural network models include RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), and Transformer (network architecture based on self-attention mechanism).

[0067] In an embodiment of the present application, the preset timing model is preferably obtained by training an LSTM neural network model, that is, the preset timing model is an LSTM (Long Short-Term Memory) neural network model obtained by pre-training using the historical storage parameter information of the data cluster and the actual situation of the historical storage capacity water level.

[0068] The time series model trained with the LSTM neural network model can effectively address the vanishing and exploding gradient issues that occur in some time series neural networks (such as RNNs), and can capture longer-term dependencies. Specifically, the preset time series model is an LSTM (Long Short-Term Memory) neural network model pre-trained using historical storage parameter information and historical storage capacity levels of the data cluster.

[0069] Vanishing gradients and exploding gradients are two common and serious problems in neural network training. Both are closely related to the gradient update mechanism during backpropagation and can affect the learning ability and training stability of neural networks, especially in deep networks and recurrent neural networks. If the gradient becomes extremely small, the network's weight updates will become very slow or even stop completely. If the gradient value is too large, the model's weight updates will become too rapid, and the network may not converge during training, resulting in unsuccessful model training.

[0070] Among them, in some possible embodiments, it can be as follows Figure 2 As shown, the preset time series model can be trained through the following steps:

[0071] S21. Obtain historical storage parameter information and historical storage capacity level information of the data cluster;

[0072] S22: Construct a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level;

[0073] S23, inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold;

[0074] S24. Determine the LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold as the preset time series model.

[0075] The process of executing step S21 can refer to the execution process of step S11, obtain the historical storage parameter information of the data cluster, and then calculate the true value of the historical capacity water level under the historical storage parameter based on the capacity situation in the historical storage parameter.

[0076] Furthermore, step S22 is executed to construct training sample data based on the historical storage parameters and the corresponding historical storage capacity water level. Similar to other neural network model training processes, the training sample data can be divided into a validation set and a test set, and then the LSTM neural network model is trained in stages using the validation set and the test set.

[0077] Furthermore, during the execution of step S12 or step S22, that is, during the model training phase and the model application phase, the same data feature construction method is adopted. Specifically, the feature construction of the time series model input can be performed through the following steps:

[0078] Data cleaning is performed on the capacity of the data cluster and the flow of the data cluster, and a storage parameter information feature matrix is ​​constructed.

[0079] As an example, Figure 3 As shown in the figure, through data collection, we can obtain daily historical capacity and traffic information. This includes: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of user storage containers, the actual usage of management storage containers, the quota capacity of user storage containers, the newly added quota capacity requested by users, the optimized quota capacity requested by users, the read and write traffic of user storage containers, and the request volume of user storage containers.

[0080] Among them, historical capacity and historical traffic can be counted as historical original data, which can be Figure 3 As shown, the original historical data is cleaned by missing value processing, data deduplication, data standardization, etc. This is to repair the errors, missing, duplicate, inconsistent or non-compliant data in the original data, thereby improving the data quality and providing reliable input for subsequent analysis, modeling or decision-making.

[0081] Then, if Figure 3 As shown, feature engineering is constructed based on the structure of data cleaning, and feature extraction, feature selection, and feature combination are performed. Specifically, a sliding window feature can be generated for the statistically obtained and cleaned historical original data, and the sliding feature may include storage parameter information for a period of time in the past (if it is at the daily level, it can be the past N days). Storage parameter information of the order of hours, days, weeks, months, etc. is extracted according to different time intensities, and then relevant time features are generated, such as peak time and periodic features. Finally, interactive features can be constructed by combining multiple fields in the extracted features to enhance the expressiveness of the LSTM neural network model. Then, based on the constructed data features, a storage parameter information matrix is ​​constructed, and then the LSTM neural network model is trained based on the storage parameter information matrix. Among them, it can be as follows Figure 3As shown, a data set is constructed based on the storage parameter information matrix, and a training set, a validation set, and a test set are constructed. The training set, the validation set, and the test set are used for LSTM modeling and model training.

[0082] That is, step S23 is further executed to input the constructed data features into the LSTM neural network model, and the time series data of the data features are converted into the data format required by LSTM, that is, the time series data is formatted according to the data format of [samples, timestamps, features], where samples is the number of samples, timesteps is the time step, and features is the number of features.

[0083] In the embodiment of the present application, the network structure of the LSTM neural network model includes: an input layer, 1 to 3 LSTM layers, a Dropout layer, and a fully connected output layer, wherein the Dropout layer can prevent data overfitting. In the process of executing step S23, the mean square error MSE (Mean Quared Error) can be used as the loss function, and the Adam optimizer can be used to perform back-propagation training on the LSTM, and the early stopping mechanism can be used to prevent overfitting. The performance of the model is monitored using a validation set, and the trained LSTM neural network model is finally evaluated on the test set to verify the availability and accuracy of the trained LSTM neural network model, until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than the preset data difference threshold. Among them, the preset data difference threshold can be flexibly set according to the actual model accuracy requirements, and this application does not strictly limit it.

[0084] Further, step S24 is executed to determine the LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold as an available time series model, that is, as a preset time series model, and then deploy the preset time series model to the execution subject of the method provided in this application, and the execution subject runs the preset time series model to execute the above step S12 to output the storage water level prediction result.

[0085] Based on the same data processing logic of step S22, the process of executing step S12 can be implemented by the following steps:

[0086] Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function;

[0087] Based on the storage water level prediction probability, a storage water level prediction result of the data cluster is determined.

[0088] As mentioned above, LSTM can perform data processing based on the input storage parameter feature matrix. The specific data processing process includes: learning the nonlinear features in the storage parameter information features based on the ReLU activation parameters, and using the linear activation function to output the storage water level prediction probability.

[0089] Specifically, ReLU (Rectified Linear Unit) is a rectified linear unit, whose mathematical expression is f(x) = max(0,x). It is computationally simple, converges quickly, and effectively alleviates the vanishing gradient problem. When the input x is greater than 0, the derivative of the function is always 1. This allows the network to more effectively transfer gradients during backpropagation, thereby accelerating model training.

[0090] In the internal structure of LSTM, such as the calculation process of the input gate, forget gate, and output gate, the ReLU activation function can perform nonlinear transformations on the input information, allowing the model to better capture complex patterns in the data. In the embodiment of the present application, when processing the time series data of storage capacity water level, the fluctuation of storage capacity water level is affected by multiple factors and presents complex nonlinear relationships. The ReLU activation function can help the LSTM model learn these relationships, thereby better predicting the future trend of storage capacity water level.

[0091] Furthermore, in the cluster water level prediction task, it is hoped that the output of the model is a continuous value between 0 and 1, which can represent a certain proportion or probability of the water level. For example, 0 can indicate that the water level is extremely low, and 1 can indicate that the water level has reached the upper limit. Using a linear activation function alone cannot directly limit the output to between 0 and 1. It may be necessary to combine it with other processing steps, such as using a Sigmoid function (whose expression is f(x) = 1 / (1+e -x1 ), which can map the input to the (0,1) interval) to achieve this goal.

[0092] In the embodiment of the present application, the different numerical values ​​obtained by the Sigmoid function mapping correspond to the storage water level prediction probability. The larger the value, the higher the future storage water level and the greater the risk. Among them, the storage water level prediction results can be simply divided into: low risk, medium risk, high risk, or early warning risk probability values, such as 10%, 50%, 80%, etc., and the storage water level prediction probability value can be determined according to the pre-predicted probability value classification rules to determine which classification rule the storage water level prediction probability value belongs to, thereby determining the storage water level prediction result to which it belongs. Figure 3 As shown, the classification model determines whether capacity expansion or contraction is needed based on the predicted probability value of the storage water level.

[0093] For example, assuming that the value output by Sigmoid is greater than 0.5, the storage water level prediction result can be determined as a high risk. In the embodiment of the present application, the processing plan corresponding to different storage water levels can be determined as a capacity management plan in advance, and then a storage water level prediction result-capacity management plan mapping relationship table is constructed. Based on this, step S13 is executed, and the capacity management plan corresponding to the high-risk storage water level prediction result can be determined as the target capacity management plan according to the storage water level prediction result-capacity management plan mapping relationship table. Figure 3 As shown in the figure, a generative model is used to determine the storage capacity level of the predicted data cluster and the number of machines to be expanded or reduced.

[0094] As an example, when the storage water level prediction result is high risk, the corresponding capacity management plan is expansion. Based on the storage water level prediction probability value, the specific number of expansion Buckets required and the specific deployment location are determined. Based on the number of Buckets that need to be expanded and the specific deployment location, the available number of Buckets and the corresponding locations are determined, and the available Buckets are used to expand the data cluster or data service whose storage water level prediction result is high risk.

[0095] To determine whether a data cluster needs to be expanded, the LSTM output can be mapped to the data interval [0, 1] using a Sigmund activation function. A threshold (for example, 0.5) is set. If it is greater than 0.5, expansion is required. To predict the required capacity, a regression model can be used to directly calculate the required resources. For data cluster capacity prediction, the LSTM model outputs the total capacity and allocated capacity for future time steps. By comparing these with the actual capacity, the health of the data cluster is dynamically assessed. Ultimately, it can automatically provide expansion and contraction solutions, predict user needs, and monitor capacity levels, providing data-driven support for data cluster managers.

[0096] On the contrary, if the predicted result of the storage water level is low risk, or the predicted probability value of the storage water level is extremely low, you can consider shrinking the data cluster. Based on the size of the predicted probability value of the storage water level, you can further determine the number of buckets that need to be shrunk and their specific deployment locations. Then, you can optimize the buckets that need to be shrunk. You can achieve the shrinking effect by pausing or completely terminating the use of some buckets.

[0097] In some possible embodiments, the method is applied to an object-based data cluster. An object-based data cluster is a distributed storage cluster designed specifically for unstructured data (such as images, videos, logs, documents, etc.). By encapsulating the data to be stored as "objects" for management, it provides a flexible, elastic, and highly scalable solution for storing, accessing, and managing massive amounts of data. Common types of object-based data clusters include AWS3, BOS, and OSS, corresponding to object storage clusters from different vendors.

[0098] Based on this, in some possible embodiments, the execution process of the above step S13 may further include the following steps:

[0099] Respond to data flow requests based on the address of the expanded or reduced storage container.

[0100] In an embodiment of the present application, the objects in the data cluster of the object storage are composed of three parts: data content, object key, and metadata. Among them, the object key is a globally unique identifier of the data, which is used to locate the object. The metadata is used to describe the properties of the object, and can be quickly retrieved and managed through the metadata.

[0101] Based on this, when executing step S13, after determining the corresponding target capacity management plan, the storage capacity of the data cluster to be processed, especially the object storage data cluster, will be expanded or reduced, and the storage capacity will be increased or decreased. At this time, for the target capacity management plan for expansion, it is necessary to migrate part of the traffic requests to the newly added capacity. For the target capacity management plan for reduction, it is necessary to migrate the traffic requests originally deployed on the capacity that needs to be reduced to the capacity that does not need to be reduced. In this process, the data content in the storage container is migrated to the target container based on the object key and metadata of the object storage, and then the metadata of the storage container to which the migrated data content belongs is updated so that the data request can accurately read, add, delete, and write the corresponding data content based on the updated metadata.

[0102] Based on the description of the storage capacity water level risk management method provided in the first aspect, in the second aspect, an embodiment of the present application provides a storage capacity water level risk management device, wherein Figure 4 As shown, the device 40 includes:

[0103] Input module 401 is used to obtain various storage parameter information of the data cluster to be processed, wherein the storage parameter information includes: the capacity of the data cluster and the flow of the data cluster;

[0104] Prediction module 402 is configured to input each of the storage parameter information into a preset time series model, and the time series model performs data processing based on the input storage parameter information to output a storage water level prediction result for the data cluster; wherein the preset time series model is an LSTM neural network model pre-trained using the historical storage parameter information of the data cluster and the actual historical storage capacity water level;

[0105] The management module 403 is used to determine a target capacity management plan based on the storage water level prediction result and execute the target capacity management plan, wherein the target capacity management plan includes: expansion or reduction, the capacity required for the expansion or the capacity required for the reduction.

[0106] In conjunction with the second aspect, in a second possible embodiment, the apparatus 40 further includes a model training module 404, wherein the model training module 404 is configured to:

[0107] Obtain historical storage parameter information and historical storage capacity level information of the data cluster;

[0108] Constructing a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level real situation;

[0109] Inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold;

[0110] The LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold is determined as the preset time series model.

[0111] In combination with the second aspect, in a third possible embodiment, the capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the new quota capacity applied for by the user, and the optimized quota capacity applied for by the user; the traffic situation of the data cluster includes but is not limited to: the read and write traffic situation of the user storage container and the request volume of the user storage container.

[0112] In conjunction with the second aspect, in a fourth possible embodiment, the prediction module is further configured to:

[0113] Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function;

[0114] Based on the storage water level prediction probability, a storage water level prediction result of the data cluster is determined.

[0115] In combination with the second aspect, in a fifth possible embodiment, the apparatus is applied to a data cluster of object storage, and the apparatus 40 further includes:

[0116] The request migration module 405 is configured to respond to the data flow request according to the address of the storage container after expansion or contraction.

[0117] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0118] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0119] In a third aspect, exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, wherein the computer program, when executed by the at least one processor, causes the electronic device to perform a method according to an embodiment of the present application.

[0120] An exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0121] An exemplary embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to cause the computer to perform the method according to the embodiment of the present application.

[0122] refer to Figure 5, a block diagram of an electronic device 500 that can serve as a server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0123] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM 502) or a computer program loaded from a storage unit 508 into a random access memory (RAM 503). In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output interface (I / O interface 505) is also connected to the bus 504.

[0124] Multiple components within electronic device 500 are connected to I / O interface 505, including an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. Input unit 506 can be any type of device capable of inputting information into electronic device 500. Input unit 506 can receive input numeric or character information and generate key input signals related to user settings and / or function control of the electronic device. Output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. Storage unit 508 may include, but is not limited to, a magnetic disk or an optical disk. Communication unit 509 allows electronic device 500 to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver and / or chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0125] The computing unit 501 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the aforementioned storage capacity water level risk management method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the aforementioned storage capacity water level risk management method by any other appropriate means (e.g., by means of firmware).

[0126] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A storage capacity water level risk management method, characterized in that: The method comprises: Acquire various storage parameter information of the data cluster to be processed, wherein the storage parameter information includes: the capacity of the data cluster and the flow of the data cluster; Inputting each of the storage parameter information into a preset time series model, the time series model performs data processing based on the input storage parameter information, and outputs a storage water level prediction result for the data cluster; wherein the preset time series model is an LSTM neural network model pre-trained using the historical storage parameter information of the data cluster and the actual historical storage capacity water level; Based on the storage water level prediction result, a target capacity management plan is determined and executed, wherein the target capacity management plan includes: expansion or reduction, the capacity required for the expansion or the capacity required for the reduction.

2. The method according to claim 1, characterized in that The preset time series model is pre-trained through the following steps: Obtain historical storage parameter information and historical storage capacity level information of the data cluster; Constructing a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level real situation; Inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold; The LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold is determined as the preset time series model.

3. The method according to claim 1, characterized in that The capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the newly added quota capacity applied for by the user, and the optimized quota capacity applied for by the user; the traffic of the data cluster includes but is not limited to: the read and write traffic of the user storage container and the request volume of the user storage container.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Performing data cleaning on the capacity of the data cluster and the flow of the data cluster, and constructing a storage parameter information feature matrix; Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function; Based on the storage water level prediction probability, a storage water level prediction result of the data cluster is determined.

5. The method according to claim 1, wherein The method is applied to a data cluster of object storage, and the method further includes: Respond to data flow requests based on the address of the expanded or reduced storage container.

6. A storage capacity water level risk management device, characterized in that: The device comprises: An input module is used to obtain various storage parameter information of the data cluster to be processed, wherein the storage parameter information includes: the capacity of the data cluster and the flow of the data cluster; A prediction module is configured to input each of the storage parameter information into a preset time series model, and the time series model performs data processing based on the input storage parameter information to output a storage water level prediction result for the data cluster; wherein the preset time series model is an LSTM neural network model pre-trained using the historical storage parameter information of the data cluster and the actual historical storage capacity and water level; A management module is used to determine a target capacity management plan based on the storage water level prediction result and execute the target capacity management plan, wherein the target capacity management plan includes: expansion or reduction, the capacity required for the expansion or the capacity required for the reduction.

7. The device according to claim 6, characterized in that The device further includes a model training module, wherein the model training module is configured to: Obtain historical storage parameter information and historical storage capacity level information of the data cluster; Constructing a training sample data set based on the historical storage parameter information and the corresponding historical storage capacity water level real situation; Inputting the training sample data set into an initial LSTM neural network model, and performing backpropagation training on the LSTM neural network model until the target data difference between the storage capacity water level prediction value output by the LSTM neural network model and the actual historical storage capacity water level is less than a preset data difference threshold; The LSTM neural network model corresponding to the target data difference being less than the preset data difference threshold is determined as the preset time series model.

8. The device according to claim 6, characterized in that The capacity of the data cluster includes but is not limited to: the total cluster capacity of the data cluster, the allocated capacity of the data cluster, the actual usage of the user storage container, the actual usage of the management storage container, the quota capacity of the user storage container, the newly added quota capacity requested by the user, and the optimized quota capacity requested by the user; the traffic of the data cluster includes but is not limited to: the read and write traffic of the user storage container and the request volume of the user storage container; The prediction module is further configured to: Inputting the storage parameter information feature matrix into the preset time series model, having the time series model learn the nonlinear features in the storage parameter information features based on ReLU activation parameters, and having the time series model output the storage water level prediction probability using a linear activation function; Determining a storage water level prediction result of the data cluster based on the storage water level prediction probability; The device is applied to a data cluster of object storage, and the device further includes: The request migration module is used to respond to data flow requests based on the address of the storage container after expansion or reduction.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing a program; wherein the program comprises instructions, and when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images