Storage volume elastic capacity expansion method and device, equipment and medium

Abstract

The invention discloses a storage volume elastic capacity expansion method and device, equipment and a medium, and relates to the technical field of storage volume capacity expansion, and the method comprises the steps: obtaining a real-time state vector of a target storage volume after data preprocessing; inputting the real-time state vector into a target storage volume capacity demand prediction model, and predicting the capacity demand in a future target long-term time based on the real-time state vector through the target storage volume capacity demand prediction model to obtain a target long-term capacity demand trend prediction value; wherein the target storage volume capacity demand prediction model is a model which learns a mapping relationship among a storage volume basic index, a time sequence mode, a service association feature and a future capacity demand; and according to the target long-term capacity demand trend predicted value, a preset capacity expansion strategy parameter and the target short-term capacity demand trend, judging whether the target storage volume executes capacity expansion operation or not. Elastic capacity expansion is automatically triggered before the capacity is used up, and stable operation of services is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of storage volume expansion, and in particular to a method, apparatus, device, and medium for elastically expanding storage volumes. Background Art

[0002] In the cloud-native era, Kubernetes (K8S, an open source platform for managing containerized applications across multiple hosts in a cloud platform) has become the de facto standard for container orchestration. Enterprises are widely deploying containerized applications on Kubernetes, using persistent storage volumes to store data. However, the traditional approach involves pre-registering and mounting volumes of specific sizes from cloud storage. Because it's difficult to accurately estimate application capacity requirements, mounting large volumes can waste resources, while mounting smaller volumes often requires frequent downtime for expansion due to insufficient capacity, impacting business stability. Kubernetes's storage resource management strategy primarily relies on manual expansion based on the experience of operations personnel and real-time monitoring, as well as automatic expansion based on fixed thresholds. This is inefficient for large clusters and complex business scenarios and can easily lead to insufficient storage capacity due to untimely responses, impacting business continuity. Automatic expansion based on fixed thresholds, unable to adapt to the complex and dynamic changes in storage requirements, often results in resource waste due to premature expansion, or service failures due to delayed expansion. Furthermore, manual capacity expansion cannot proactively detect capacity issues. Users can only address them after storage volumes run out of capacity, impacting normal business operations. This prevents proactively predicting capacity crises and makes it difficult to meet the ever-changing storage needs of businesses. Meanwhile, artificial intelligence (AI) technology has achieved remarkable results in data analysis and prediction. With the rise of applications such as AI (Artificial Intelligence) models, storage resource demands are extremely high, and traffic fluctuates like tidal waves. This makes integrating AI with elastic storage volume expansion in Kubernetes environments, to meet dynamic business storage needs through automation and intelligence, a critical issue that needs to be addressed urgently. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a method, apparatus, device and medium for elastic expansion of storage volumes, which can solve the problem of elastic expansion of storage volumes in a K8S cluster.

[0004] To solve the above technical problems, an embodiment of the present invention provides a method for elastically expanding a storage volume, comprising: Obtaining a real-time state vector of the target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement; The real-time state vector is input into a target storage volume capacity demand prediction model, so that the target storage volume capacity demand prediction model predicts the capacity demand in the future target long-term period based on the real-time state vector to obtain a target long-term capacity demand trend forecast value; wherein the target storage volume capacity demand prediction model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics, and future capacity demand; Determine whether to perform capacity expansion on the target storage volume based on the target long-term capacity demand trend forecast value, preset capacity expansion policy parameters, and target short-term capacity demand trend.

[0005] Optionally, obtain the real-time state vector of the target storage volume after data preprocessing, including: Collect the initial status data of the cluster's target storage volume by calling cluster test cases and the storage system's management interface; The initial state data is preprocessed to obtain a real-time storage volume basic indicator vector, a real-time operation indicator vector, and a real-time business-related indicator vector of the target storage volume.

[0006] Optionally, before inputting the real-time state vector into the target storage volume capacity demand prediction model, the following steps are further included: Regularly collect historical status data of each storage volume in the cluster; historical status data includes historical storage volume basic indicators, historical operation indicators, and historical business-related indicators; Performing cleaning processing on the historical state data to obtain cleaned historical state data after removing noise data, duplicate data and error data, and converting the cleaned historical state data into numerical data in a numerical format; Aggregate and normalize numerical data according to preset time windows to obtain historical storage volume basic characteristics, historical time series characteristics, and historical business correlation characteristics; Extracting target historical storage volume basic features, target historical time series features, and target historical business association features related to storage volume capacity requirements from historical storage volume basic features, historical time series features, and historical business association features to obtain a feature training dataset; The feature training dataset is used to train the initial model built based on the long short-term memory neural network to obtain the target storage volume capacity demand prediction model.

[0007] Optionally, the initial model includes an input layer, an LSTM hidden layer, a Dropout layer, and an output layer; Accordingly, the feature training dataset is used to train the initial model built based on the long short-term memory neural network to obtain a target storage volume capacity demand prediction model, including: Input each target historical feature in the feature training dataset into the input layer so that the input layer can convert the feature training dataset into a target feature training tensor according to the set target dimension, and pass the target feature training tensor to the LSTM hidden layer; Capture the long-term dependencies of target feature training tensors through the LSTM mechanism of the LSTM hidden layer; The output tensor of the connected LSTM hidden layer is randomly discarded through the Dropout layer to obtain the discarded feature representation, and the discarded feature representation output by each Dropout layer is fully connected to obtain the target feature representation; The target feature representation is mapped to the capacity demand trend prediction value through the output layer, and the model parameters of the initial model are reversely optimized based on the capacity demand trend prediction value and the capacity demand trend label value until the preset training end conditions are met. The model under the current model parameters is then used as the target storage volume capacity demand prediction model.

[0008] Optionally, determining whether to perform a capacity expansion operation on the target storage volume based on the target long-term capacity demand trend forecast value, preset capacity expansion policy parameters, and the target short-term capacity demand trend includes: If the target long-term capacity demand trend forecast value is greater than the preset expansion threshold, then checking whether the target short-term capacity demand trend is consistent with the long-term capacity demand trend of the target long-term capacity demand trend forecast value; If they are consistent, the target storage volume is determined to perform the expansion operation.

[0009] Optionally, after determining whether to perform the expansion operation on the target storage volume, the method further includes: Sending a capacity expansion request including the increased storage capacity to the storage system, so that the storage system allocates capacity to the target storage volume based on the capacity expansion request; The target storage volume is expanded according to the expansion scale, expansion time window, and expansion method parameters of the preset expansion policy parameters.

[0010] Optionally, the storage volume elastic expansion method also includes: The actual storage volume data and actual business operation information after the target storage volume expansion is completed are compared and analyzed with the target long-term capacity demand trend forecast value. Based on the comparative analysis results, the target storage volume capacity demand forecast model is incrementally trained to obtain an updated target storage volume capacity demand forecast model.

[0011] In a second aspect, an embodiment of the present invention discloses a storage volume elastic expansion device, comprising: A vector acquisition module is used to obtain a real-time state vector of the target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement; A demand trend forecasting module is configured to input the real-time state vector into a target storage volume capacity demand forecasting model, so as to use the target storage volume capacity demand forecasting model to forecast the capacity demand in the future target long-term period based on the real-time state vector, thereby obtaining a target long-term capacity demand trend forecast value; wherein the target storage volume capacity demand forecasting model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics, and future capacity demand; The capacity expansion determination module is used to determine whether to perform a capacity expansion operation on the target storage volume based on the target long-term capacity demand trend forecast value, preset capacity expansion policy parameters, and target short-term capacity demand trend.

[0012] In a third aspect, an embodiment of the present invention discloses an electronic device, including: memory for storing computer programs; A processor is used to execute a computer program to implement the steps of the storage volume elastic expansion method disclosed above.

[0013] In a fourth aspect, an embodiment of the present invention discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for elastically expanding storage volumes are implemented.

[0014] It can be seen that the present application discloses a method for elastic expansion of storage volumes, including: obtaining a real-time state vector of a target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity demand; inputting the real-time state vector into a target storage volume capacity demand prediction model, so as to predict the capacity demand in the future target long-term time based on the real-time state vector through the target storage volume capacity demand prediction model, so as to obtain a target long-term capacity demand trend prediction value; wherein the target storage volume capacity demand prediction model is a model that has learned the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics and future capacity demands; and determining whether the target storage volume performs an expansion operation based on the target long-term capacity demand trend prediction value, preset expansion strategy parameters, and target short-term capacity demand trend.

[0015] As can be seen from the above technical solution, by collecting and preprocessing the real-time state vector of the target storage volume, the long-term capacity demand can be accurately predicted with the help of the target storage volume capacity demand prediction model and the real-time state vector. Then, the long-term capacity demand and the short-term capacity demand forecast values ​​and the expansion strategy parameters can be combined to accurately estimate the capacity demand for a period of time in the future, thereby forming a targeted elastic expansion plan, effectively avoiding resource waste and frequent manual expansion, and significantly reducing storage costs. Through real-time monitoring, it can automatically trigger elastic expansion before capacity is exhausted, ensuring stable business operation, eliminating the risk of downtime caused by manual operations, and improving business continuity and availability. At the same time, the system realizes automated and intelligent expansion, reducing the burden of operation and maintenance, reducing the rate of human errors, and customizing expansion strategies based on business characteristics to improve cluster management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A flow chart of a method for elastically expanding storage volumes provided by an embodiment of the present invention; Figure 2 A flowchart of a method for elastically expanding Kubernetes storage volumes based on artificial intelligence provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a storage volume elastic expansion device provided by an embodiment of the present invention; Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0020] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] In the cloud-native era, Kubernetes has become the de facto standard for container orchestration. Enterprises widely containerize and deploy applications in Kubernetes, using persistent storage volumes to store data. However, the traditional approach is to pre-apply for and mount storage volumes of a specific size in cloud storage. Because it's difficult to accurately estimate the capacity requirements of applications, mounting large volumes can easily lead to resource waste. Mounting small volumes, however, requires frequent downtime and expansion due to insufficient capacity, impacting business stability. Manual expansion also fails to detect capacity issues in advance. At the same time, artificial intelligence technology has achieved remarkable results in data analysis and prediction. With the rise of applications such as large AI models, storage resource demands are extremely high, and traffic fluctuates in a tidal manner. This makes it a key issue that needs to be addressed in a Kubernetes environment: combining artificial intelligence to achieve elastic expansion of storage volumes, and to meet the dynamic storage needs of businesses through automation and intelligence.

[0022] Kubernetes's storage resource management strategy primarily relies on the experience of operations and maintenance personnel and real-time monitoring for manual capacity expansion and automatic capacity expansion based on fixed thresholds. This is not only inefficient when faced with large-scale clusters and complex business scenarios, but can also easily lead to insufficient storage capacity due to untimely responses, impacting business continuity. Automatic capacity expansion based on fixed thresholds, unable to adapt to the complex and dynamic changes in storage requirements, often results in problems such as premature capacity expansion leading to resource waste, or late capacity expansion leading to service failures.

[0023] While Kubernetes can manage large-scale container clusters through its own resource management strategies, manual capacity expansion relies on the experience and real-time monitoring of operations personnel. This is not only inefficient for large clusters and complex business scenarios, but can also lead to insufficient storage capacity due to untimely responses, impacting business continuity. Automatic capacity expansion based on fixed thresholds, unable to adapt to the complex and dynamic changes in storage requirements, often results in problems such as premature capacity expansion leading to resource waste or delayed capacity expansion leading to service failures. Some existing AI-based capacity expansion solutions lack the ability to handle both localized mutations in stored data and long-term trends, making it difficult to accurately and timely meet the elastic capacity expansion needs of storage volumes.

[0024] In a Kubernetes environment, traditional storage volume management models face numerous challenges that need to be overcome. On the one hand, estimating the capacity requirements of persistent volume storage for application containers is extremely difficult. Mounting larger volumes results in idle and wasted valuable storage resources. Mounting smaller volumes, on the other hand, frequently encounters capacity shortages, forcing users to downtime and expand capacity, which undoubtedly poses a serious threat to business continuity and stability. On the other hand, the current manual capacity expansion mechanism has significant lags. Users can only address this issue after the storage volume capacity is exhausted, impacting the normal operation of the business. This makes it impossible to proactively predict capacity crises and makes it difficult to meet the ever-changing storage needs of the business. Furthermore, with the rise of emerging applications such as large AI models, which present high demands for storage resources and tidal traffic characteristics, further exacerbating the complexity of storage management.

[0025] To this end, the present invention provides a storage volume elastic expansion solution that can solve the elastic expansion problem of storage volumes in a K8S cluster.

[0026] Reference Figure 1 As shown, the present invention discloses a method for elastically expanding storage volumes, comprising: Step S11: obtaining a real-time state vector of the target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement.

[0027] In this embodiment, the initial state data of the target storage volume of the cluster is collected by calling cluster test cases and the storage system's management interface; the initial state data is preprocessed to obtain the target storage volume's real-time storage volume basic indicator vector, real-time operation indicator vector, and real-time business-related indicator vector. It is understood that the data collection module, based on the Prometheus Exporter mechanism, calls the Kubernetes API and the storage system's management interface to regularly collect initial state data such as basic storage volume information (such as storage volume type and container), operation data (used capacity, remaining capacity, data read and write rate, I / O (Input / Output) request count), and application business data (business type, access traffic trends). The data processing module then preprocesses the initial state data. Specifically, the collected initial state data is cleaned of anomalies, such as using a Z-score to remove transient noise, duplicates, and erroneous data. Data in different formats is converted to a standard format and aggregated according to time windows (e.g., every 5 minutes). Numerical data is normalized using normalization methods to generate preprocessed initial state data, improving the accuracy and efficiency of subsequent data analysis and model training. The feature extraction module extracts state vectors related to storage volume capacity requirements from the preprocessed initial state data. These state vectors include real-time storage volume basic indicator vectors (e.g., used capacity percentage, average read / write rate), real-time operational indicator vectors (e.g., data operational changes), and real-time business-related indicator vectors (e.g., the impact of peak business hours on storage demand). Feature selection algorithms (e.g., correlation coefficient method) are used to identify key features and construct the real-time state vector.

[0028] Step S12: Input the real-time state vector into the target storage volume capacity demand prediction model, so as to predict the capacity demand in the future target long-term time based on the real-time state vector through the target storage volume capacity demand prediction model, so as to obtain the target long-term capacity demand trend prediction value; wherein, the target storage volume capacity demand prediction model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics and future capacity demand.

[0029] In this embodiment, before the real-time state vector is input into the target storage volume capacity demand prediction model, it also includes: regularly collecting historical status data of each storage volume of the cluster; the historical status data includes historical storage volume basic indicators, historical operation indicators and historical business-related indicators; performing cleaning processing on the historical status data to obtain cleaned historical status data after removing noise data, duplicate data and error data, and converting the cleaned historical status data into numerical data in a numerical format; aggregating and normalizing the numerical data according to a preset time window to obtain historical storage volume basic features, historical time series features and historical business-related features; extracting target historical storage volume basic features, target historical time series features and target historical business-related features related to storage volume capacity requirements from the historical storage volume basic features, historical time series features and historical business-related features to obtain a feature training data set; it can be understood that the historical status information of each storage volume of the K8S cluster is regularly collected through the data collection module, where the information collection process and collection mechanism are the same as the initial state data, and no further details are given. It should be noted that historical status data includes historical storage volume basic indicators, historical operation indicators, and historical business-related indicators. Among them, historical storage volume basic indicators include the percentage of stored volume usage capacity, average read and write rates, etc., while historical operation indicators are operational data generated during system operation, such as: storage volume I / O queue length, error rate, disk utilization, etc., which are used to obtain historical time series characteristics from historical operation indicators. Historical time series characteristics include data change trends, cyclical fluctuations, etc. Historical business-related indicators include the impact of busy business periods on storage demand. The above historical status data is then cleaned. The cleaning process is the same as the cleaning process for the initial status data and will not be repeated here. The cleaned historical status data is then formatted to obtain digitized numerical data. Specifically, the non-numeric data in the cleaned historical status data is digitized and processed into a numerical form that can be processed by the model. For example, the business peak period in the text feature of the cleaned historical status data is encoded as [1,0,0], and the storage volume type in the classification feature is mapped to 0 / 1; the time feature, such as 2023-06-15 10:00, is converted into a timestamp or periodic feature, and then the numerical data is aggregated according to a 5-minute time window to reduce noise and calculate the average and maximum values ​​within the time window. The resource usage of multiple related Pods (atomic scheduling units and container design patterns) is then merged into a storage volume-level load indicator. It should be noted that during the merging process, features under different dimensions are normalized to obtain a feature training dataset.

[0030] In this embodiment, a feature training data set is used to perform model training on an initial model constructed based on a long short-term memory neural network to obtain a target storage volume capacity demand prediction model. It can be understood that the initial model includes an input layer, an LSTM (Long Short-Term Memory) hidden layer, a Dropout layer, and an output layer; accordingly, the initial model constructed based on the long short-term memory neural network is trained using a feature training dataset to obtain a target storage volume capacity demand prediction model, including: inputting the corresponding historical features of each target in the feature training dataset into the input layer, so that the input layer converts the feature training dataset into a target feature training tensor according to the set target dimension, and passes the target feature training tensor to the LSTM hidden layer; capturing the long-term dependency of the target feature training tensor through the LSTM mechanism of the LSTM hidden layer; randomly discarding the output tensor of the connected LSTM hidden layer through the Dropout layer to obtain a discarded feature representation, and fully connecting the discarded feature representations output by each Dropout layer to obtain a target feature representation; mapping the target feature representation to a capacity demand trend prediction value through the output layer, and reversely optimizing the model parameters of the initial model based on the capacity demand trend prediction value and the capacity demand trend label value until the preset training end conditions are met, and then the model under the current model parameters is used as the target storage volume capacity demand prediction model. It's understandable that the input layer dimensions match those of the feature dataset. For example, if the feature dataset includes five metrics, such as percentage of used capacity, read / write rates, and temporal features, the input layer will have five neurons, converting the input features into target feature training tensors. The input layer receives preprocessed time series feature data (such as normalized historical capacity data and business metrics) and passes the resulting target feature training tensor to the first LSTM hidden layer. The input data is formatted as a three-dimensional tensor [batch_size, time_steps, feature_dim], where time_steps represents the length of the time window (e.g., data points from the past 24 hours). The LSTM hidden layer consists of at least two LSTM layers, each containing 50-100 LSTM neurons. LSTM neurons capture long-term dependencies in the time series (such as cyclical fluctuations and growth trends in storage capacity) through a forget gate, input gate, and output gate mechanism. Multi-layer LSTMs extract features step by step: the bottom-level LSTM captures short-term fluctuations (such as hourly read / write changes), while the higher-level LSTM captures long-term trends (such as weekly / monthly capacity growth). The first LSTM layer receives the input layer data, and the subsequent LSTM layers receive the output of the previous LSTM layer.The output of each LSTM layer is a two-dimensional tensor [batch_size, hidden_units] (number of hidden layer neurons), or the time dimension is retained for multi-layer connections, depending on whether return_sequences=True is set. A dropout layer is placed after each LSTM layer, with a dropout rate of 0.2-0.3 (i.e., randomly dropping 20%-30% of the neuron outputs). By randomly dropping neurons, co-dependencies between neurons are reduced, preventing model overfitting during training (especially for noisy time series data such as storage capacity). The dropout layer is connected directly after the LSTM layer and randomly drops the LSTM layer output before passing it to the next layer. The fully connected layer has a number of neurons equal to the prediction time step (for example, for a 12-hour forecast, the output dimension is 12). The abstract features extracted by the LSTM layer are mapped into specific capacity prediction values. The fully connected layer generates the final prediction result through a weighted summation and an activation function (typically a linear activation, as capacity prediction is a regression task). The fully connected layer receives the output of the last dropout layer and maps it to the output dimension via a fully connected weight matrix. The output is [batch_size, prediction_steps], which is the predicted value for each sample T time steps into the future. Through this architecture, the model transforms time series inputs such as historical capacity data and business characteristics into continuous predictions of future capacity. Combined with training optimization (such as the Adam optimizer and early stopping), it achieves high-precision dynamic predictions. It is important to note that the model's LSTM layer enables cross-feature correlation learning. Specifically, when the service busyness indicator increases, the LSTM uses an input gate to increase its focus on read and write rate features. It also considers historical trends in capacity utilization to determine the impact of business peaks on capacity growth. The forget gate automatically filters out irrelevant features based on historical data (such as occasional read and write fluctuations) while retaining features that are critical for capacity prediction (such as the positive correlation between consistently increasing business traffic and capacity utilization). Finally, the underlying LSTM captures short-term fluctuations in single features (such as hourly variations in read and write rates), while the higher-level LSTM integrates long-term correlations across multiple features (such as the cyclical relationship between quarterly business peaks and sudden capacity increases).

[0031] In addition, current model training is supervised learning, which means that the training process relies on the actual storage capacity demand labels in the historical training set. The core goal is to learn the mapping relationship between feature vectors (input) and future capacity values ​​(output). Specifically: the input data includes historical time series features (such as capacity usage data for the past N time steps), basic storage volume indicators (such as total capacity, percentage of used space), and business-related features (such as business traffic, access frequency, etc.). Label data: the actual capacity demand value corresponding to the input feature (such as the actual capacity usage in the next 1 time step). After the model generates a predicted value through forward propagation, it is necessary to calculate the loss based on the label (such as mean square error (MSE), mean absolute error (MAE), etc.). The loss function measures the deviation between the model prediction and the true value and is the optimization target. The loss function is then passed to each layer through a backpropagation algorithm (time-based backpropagation). The LSTM weights, bias, and other parameters are adjusted to make the model closer to the true value in subsequent predictions. Specifically, the weights of the forget gate, input gate, and output gate in the LSTM unit are adjusted to more accurately capture the relationship between time series features and capacity. The weights of the fully connected layer are adjusted to make the output closer to the label value. Through multiple rounds of iteration (epochs), the model continuously reduces the loss and ultimately learns the mapping between features and capacity.

[0032] In this embodiment, after obtaining the trained target storage volume capacity demand prediction model, the real-time state vector is input into the target storage volume capacity demand prediction model. The prediction model can predict the target long-term capacity demand trend prediction value in the future target long-term time corresponding to the current real-time state vector through the intelligent prediction module based on the mapping relationship between the characteristics learned by itself and the capacity demand value in the future time.

[0033] In addition, in the intelligent prediction module, to adapt to the resource-constrained requirements of edge computing scenarios, model lightweighting technology can also be used to optimize the LSTM network. The specific implementation methods are as follows: The multi-layer LSTM network is simplified to one or two layers, and some hidden layers are replaced with a self-attention mechanism. The attention mechanism calculates weights at different time steps in the input sequence to focus on key features (such as periods of capacity surges). This is equivalent to the feature screening capability of a multi-layer LSTM, but with reduced computational effort. The self-attention mechanism focuses on key features by calculating query-key-value triples, assigning different weights to features at different time steps. For example, in capacity forecasting, the attention mechanism assigns higher weight to read and write rates during peak business periods and ignores noisy data during off-peak hours. Furthermore, linear quantization is used to quantize model parameters from 32-bit floating point numbers to 8-bit integers. Weight pruning removes redundant connections, reducing memory usage and model size. The lightweight model is deployed on edge nodes in the Kubernetes cluster, utilizing local monitoring data for real-time predictions, reducing the latency and bandwidth consumption of data transmission back to the central node. After training the model locally, the edge nodes collaborate with the central model through encrypted parameter aggregation to prevent sensitive data leakage. In this way, the lightweight model increases inference speed by 300% and reduces memory usage by 70% while keeping the prediction accuracy at no more than 5%. It is suitable for edge storage scenarios with high real-time requirements (such as IoT device log storage).

[0034] Step S13: Determine whether to perform a capacity expansion operation on the target storage volume based on the target long-term capacity demand trend prediction value, the preset capacity expansion policy parameters, and the target short-term capacity demand trend.

[0035] In this embodiment, if the target long-term capacity demand trend forecast value is greater than a preset expansion threshold, the target short-term capacity demand trend is checked to see if it is consistent with the long-term capacity demand trend of the target long-term capacity demand trend forecast value. If they are consistent, a capacity expansion operation is performed on the target storage volume. It will be appreciated that the target long-term capacity demand trend forecast value for the future period (the next 6 hours / 12 hours) predicted by the real-time state vector is obtained, and the target long-term capacity demand trend forecast value includes a single-point forecast value and a trend curve. Then, the target long-term capacity demand trend forecast value is compared with the preset expansion threshold. If the target long-term capacity demand trend forecast value is greater than the preset expansion threshold, it is further determined whether the target short-term capacity demand trend (the slope of change of data in the current sliding window) is consistent with the target long-term capacity demand trend. If they are consistent, it is determined that the target storage volume needs to perform an expansion operation; if the target short-term capacity demand trend is inconsistent with the long-term capacity demand trend, it is determined whether the prediction confidence corresponding to the current target long-term capacity demand trend forecast value exceeds the confidence threshold. If so, it is determined that the target storage volume needs to perform an expansion operation; if the target long-term capacity demand trend forecast value is less than the preset expansion threshold, the target short-term capacity demand trend is inconsistent with the long-term capacity demand trend, and the target short-term capacity demand trend changes suddenly, then the current target long-term capacity demand trend is ignored, and it is determined that the target storage volume needs to perform an expansion operation.

[0036] In this embodiment, after determining that the target storage volume is to be expanded, the method further includes: sending an expansion request including the increased storage capacity to the storage system, so that the storage system allocates capacity to the target storage volume based on the expansion request; and executing the expansion operation on the target storage volume according to the expansion scale parameter, expansion time window parameter, and expansion method parameter of the preset expansion policy parameters. It is understood that the expansion execution module receives the expansion instruction from the elasticity decision module and executes the expansion operation by calling the Kubernetes API and the storage system's CSI interface. The specific steps are as follows: First, a capacity expansion request is sent to the storage system, specifying the amount of capacity to be added. During the expansion process, the target storage volume is expanded according to the expansion scale parameter, expansion time window parameter, and expansion method parameter. The expansion scale parameter is the amount of storage capacity added each time the capacity is expanded (such as GB, TB) or the ratio. The time window parameter is the time range or trigger period for the expansion operation. The expansion method parameter is the specific implementation method of the expansion (such as vertical expansion, horizontal expansion). Then, the storage system completes operations such as capacity allocation and data migration. Finally, Kubernetes updates the relevant information of the storage volume to ensure that the application can perceive the expanded storage volume and use it normally. During the expansion process, the expansion progress is monitored in real time. When an abnormal situation occurs, such as expansion failure or data loss, the rollback operation is carried out in a timely manner, and an alarm information is sent to the administrator.

[0037] In this embodiment, the actual storage volume data and actual business operation information after the target storage volume expansion is completed are compared and analyzed with the target long-term capacity demand trend forecast value, and the target storage volume capacity demand forecast model is incrementally trained based on the comparative analysis results to obtain an updated target storage volume capacity demand forecast model. It is understandable that after the storage volume expansion is completed, the feedback optimization module collects actual storage usage data and business operation status and compares and analyzes them with the forecast results. Based on the analysis results, the model parameters of the intelligent prediction module are adjusted and optimized, and the expansion strategy is updated at the same time, so that the system can continuously adapt to the dynamic changes in the storage requirements of the Kubernetes cluster.

[0038] Reference Figure 2 As shown, the present invention proposes an artificial intelligence-based Kubernetes storage volume elastic expansion method and system, which includes: a data acquisition module, a data processing module, a feature extraction module, an intelligent prediction module, an elastic decision module, an expansion execution module, and a feedback optimization module.

[0039] Data collection module: collects multi-dimensional data of storage volumes in the Kubernetes cluster; Data processing module: cleans, converts formats, aggregates and normalizes the collected raw data; Feature extraction module: extracts and selects key features from the processed data; Intelligent Prediction Module: A deep learning algorithm is used to construct a storage volume capacity demand prediction model to predict storage volume capacity requirements. Specifically, the model is built based on deep learning algorithms (such as the Long Short-Term Memory (LSTM) network). Feature-selected data is divided into training, validation, and test sets. The model is trained on the training set, and model performance is optimized by adjusting model hyperparameters such as the learning rate and the number of hidden layer neurons. The validation set is used to monitor the training process to prevent overfitting. Finally, the model's prediction accuracy is evaluated on the test set, calculating metrics such as mean squared error (MSE) and mean absolute error (MAE) to ensure that the model meets actual application requirements. The trained model receives preprocessed monitoring data in real time to predict future storage volume capacity usage. Simultaneously, model parameters are regularly updated based on analysis results from the feedback optimization module, ensuring that the model can continuously adapt to dynamic changes in storage volume usage patterns.

[0040] Elastic decision module: used to make elastic expansion decisions based on the prediction results of the intelligent prediction module and the judgment results of the rapid detection algorithm based on the sliding average and fixed threshold, combined with the preset expansion strategy; specifically, based on the prediction results of the intelligent prediction module, combined with the preset expansion strategy and the judgment results of the rapid detection algorithm based on the sliding average and fixed threshold, make elastic expansion decisions, including whether to trigger expansion, the expansion capacity, and the expansion time window. When the sliding average value of the storage volume usage predicted by the intelligent prediction module within a certain period of time in the future is calculated by the algorithm and is greater than the warning trigger threshold or the expansion trigger threshold, the expansion operation is performed, otherwise the data information is fed back for analysis and optimization. Among them, the formula of the rapid detection algorithm based on the sliding average and fixed threshold is as follows: Sliding window definition, assuming the time series data is ,in, Indicates the storage volume at time The percentage of capacity used; Define the sliding window size as T, then at time The data in the window is: ,in, It includes arrive T data points.

[0041] Sliding average calculation, at time SMA t The calculation formula is: ; Fixed threshold definition, set two fixed thresholds: warning threshold: T warning =80%, expansion threshold: T expand =90%. Trigger condition judgment, if SMA t >T warning , warning trigger = True, otherwise False; if SMA t >T expand , Expansion trigger = True, otherwise False. Expansion capacity calculation, when the expansion condition is triggered, the new capacity after expansion is calculated as: New capacity = current capacity × (1 + expansion ratio), where the expansion ratio is usually set to 20%. Expansion execution module: used to receive expansion instructions and execute storage volume expansion operations; Feedback optimization module: used to compare actual data with prediction results and optimize the model and expansion strategy of the intelligent prediction module.

[0042] In this way, the AI-based Kubernetes storage volume elastic expansion method and system builds an efficient data collection and preprocessing module, integrating data from multiple sources, including storage volume usage, container status, and business traffic. Through cleansing and normalization, it provides high-quality data for the model. Secondly, a precise prediction model is designed, leveraging deep learning algorithms to mine data characteristics and capacity variation patterns to accurately predict storage demand. Furthermore, a pre-set expansion strategy is formulated by combining it with a fast detection algorithm based on a sliding average and fixed thresholds to guide volume expansion operations. Finally, a feedback optimization system is established to continuously iteratively optimize the prediction model and decision-making strategy based on actual expansion results, improving the timeliness and accuracy of expansion. Intelligent algorithms accurately predict capacity needs, effectively avoiding resource waste and frequent manual expansion, and significantly reducing storage costs. Through real-time monitoring, it automatically triggers elastic expansion before capacity is exhausted, ensuring stable business operations, eliminating the risk of downtime caused by manual operations, and improving business continuity and availability. Furthermore, the system implements automated and intelligent expansion, reducing the O&M burden and the rate of manual errors. It also customizes expansion strategies based on business characteristics, improving cluster management efficiency. Faced with complex business scenarios such as large AI models, the system can also quickly respond to changes in storage needs, providing solid support for corporate innovation and development.

[0043] As can be seen from the above technical solution, by collecting and preprocessing the real-time state vector of the target storage volume, the long-term capacity demand can be accurately predicted with the help of the target storage volume capacity demand prediction model and the real-time state vector. Then, the long-term capacity demand and the short-term capacity demand forecast values ​​and the expansion strategy parameters can be combined to accurately estimate the capacity demand for a period of time in the future, thereby forming a targeted elastic expansion plan, effectively avoiding resource waste and frequent manual expansion, and significantly reducing storage costs. Through real-time monitoring, it can automatically trigger elastic expansion before capacity is exhausted, ensuring stable business operation, eliminating the risk of downtime caused by manual operations, and improving business continuity and availability. At the same time, the system realizes automated and intelligent expansion, reducing the burden of operation and maintenance, reducing the rate of human errors, and customizing expansion strategies based on business characteristics to improve cluster management efficiency.

[0044] Reference Figure 3 As shown, the present invention also discloses a storage volume elastic expansion device, including: The vector acquisition module 11 is used to obtain the real-time state vector of the target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement; The demand trend prediction module 12 is configured to input the real-time state vector into a target storage volume capacity demand prediction model, so as to predict the capacity demand in the future target long-term period based on the real-time state vector using the target storage volume capacity demand prediction model to obtain a target long-term capacity demand trend prediction value. The target storage volume capacity demand prediction model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics, and future capacity demand. The expansion determination module 13 is configured to determine whether to perform an expansion operation on the target storage volume based on the target long-term capacity demand trend prediction value, preset expansion policy parameters, and the target short-term capacity demand trend.

[0045] As can be seen from the above technical solution, by collecting and preprocessing the real-time state vector of the target storage volume, the long-term capacity demand can be accurately predicted with the help of the target storage volume capacity demand prediction model and the real-time state vector. Then, the long-term capacity demand and the short-term capacity demand forecast values ​​and the expansion strategy parameters can be combined to accurately estimate the capacity demand for a period of time in the future, thereby forming a targeted elastic expansion plan, effectively avoiding resource waste and frequent manual expansion, and significantly reducing storage costs. Through real-time monitoring, it can automatically trigger elastic expansion before capacity is exhausted, ensuring stable business operation, eliminating the risk of downtime caused by manual operations, and improving business continuity and availability. At the same time, the system realizes automated and intelligent expansion, reducing the burden of operation and maintenance, reducing the rate of human errors, and customizing expansion strategies based on business characteristics to improve cluster management efficiency.

[0046] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4 This is a structural diagram of an electronic device according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the storage volume elastic expansion method disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0047] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0048] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0049] The operating system 221 is used to manage and control the hardware devices on the electronic device, as well as the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the storage volume elastic expansion method performed by the electronic device as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0050] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for elastically expanding storage volumes. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0051] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0052] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0053] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0054] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0055] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for elastically expanding storage volumes, characterized in that: include: Acquire a real-time state vector of the target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement; Inputting the real-time state vector into a target storage volume capacity demand prediction model, so as to predict the capacity demand in the future target long-term period based on the real-time state vector by the target storage volume capacity demand prediction model, so as to obtain a target long-term capacity demand trend prediction value; wherein the target storage volume capacity demand prediction model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics, and future capacity demand; Whether to perform a capacity expansion operation on the target storage volume is determined according to the target long-term capacity demand trend prediction value, the preset capacity expansion policy parameters, and the target short-term capacity demand trend.

2. The method for elastically expanding storage volumes according to claim 1, wherein: The step of obtaining a real-time state vector of the target storage volume after data preprocessing includes: Collect the initial status data of the cluster's target storage volume by calling cluster test cases and the storage system's management interface; The initial state data is pre-processed to obtain a real-time storage volume basic indicator vector, a real-time operation indicator vector, and a real-time service-related indicator vector of the target storage volume.

3. The method for elastically expanding storage volumes according to claim 1, wherein: Before inputting the real-time state vector into the target storage volume capacity demand prediction model, the method further includes: Regularly collect historical status data of each storage volume in the cluster; the historical status data includes historical storage volume basic indicators, historical operation indicators and historical business-related indicators; Performing a cleaning process on the historical state data to obtain cleaned historical state data after noise data, duplicate data, and erroneous data are deleted, and converting the cleaned historical state data into numerical data in a numerical format; Aggregating and normalizing the numerical data according to a preset time window to obtain historical storage volume basic characteristics, historical time series characteristics, and historical business correlation characteristics; Extracting target historical storage volume basic features, target historical time series features, and target historical business association features related to the storage volume capacity requirement from the historical storage volume basic features, the historical time series features, and the historical business association features to obtain a feature training data set; The feature training data set is used to perform model training on an initial model constructed based on a long short-term memory neural network to obtain a target storage volume capacity demand prediction model.

4. The method for elastically expanding storage volumes according to claim 3, wherein: The initial model includes an input layer, an LSTM hidden layer, a Dropout layer, and an output layer; Accordingly, the method of using the feature training data set to perform model training on the initial model constructed based on the long short-term memory neural network to obtain a target storage volume capacity demand prediction model includes: Input each target historical feature in the feature training data set into the input layer, so that the input layer converts the feature training data set into a target feature training tensor according to the set target dimension, and passes the target feature training tensor to the LSTM hidden layer; Capturing the long-term dependency of the target feature training tensor through the LSTM mechanism of the LSTM hidden layer; Randomly discarding the output tensor of the connected LSTM hidden layer through the Dropout layer to obtain a discarded feature representation, and performing full connection processing on the discarded feature representation output by each Dropout layer to obtain a target feature representation; The target feature representation is mapped into a capacity demand trend prediction value through the output layer, and the model parameters of the initial model are reversely optimized according to the capacity demand trend prediction value and the capacity demand trend label value until the preset training end conditions are met. The model under the current model parameters is then used as the target storage volume capacity demand prediction model.

5. The storage volume elastic expansion method according to claim 1, characterized in that: The determining whether to perform the capacity expansion operation on the target storage volume according to the target long-term capacity demand trend prediction value, the preset capacity expansion policy parameters, and the target short-term capacity demand trend includes: If the target long-term capacity demand trend forecast value is greater than the preset expansion threshold, detecting whether the target short-term capacity demand trend is consistent with the long-term capacity demand trend of the target long-term capacity demand trend forecast value; If they are consistent, it is determined that the target storage volume performs the expansion operation.

6. The method for elastically expanding storage volumes according to claim 5, wherein: After determining that the target storage volume performs the capacity expansion operation, the method further includes: Sending a capacity expansion request including an increased storage capacity size to a storage system, so that the storage system allocates capacity to a target storage volume based on the capacity expansion request; The expansion operation is performed on the target storage volume according to the expansion scale parameter, the expansion time window parameter, and the expansion method parameter of the preset expansion policy parameters.

7. The method for elastically expanding a storage volume according to any one of claims 1 to 6, wherein: Also includes: The actual storage volume data and actual business operation information after the target storage volume expansion is completed are compared and analyzed with the target long-term capacity demand trend forecast value, and the target storage volume capacity demand forecast model is incrementally trained based on the comparative analysis results to obtain an updated target storage volume capacity demand forecast model.

8. A storage volume elastic expansion device, characterized in that: include: A vector acquisition module, configured to acquire a real-time state vector of a target storage volume after data preprocessing; wherein the real-time state vector is a feature vector related to the storage volume capacity requirement; a demand trend prediction module, configured to input the real-time state vector into a target storage volume capacity demand prediction model, so as to predict the capacity demand in the future target long-term period based on the real-time state vector through the target storage volume capacity demand prediction model, thereby obtaining a target long-term capacity demand trend prediction value; wherein the target storage volume capacity demand prediction model is a model that learns the mapping relationship between storage volume basic indicators, time series patterns, business-related characteristics, and future capacity demand; The capacity expansion determination module is used to determine whether to perform a capacity expansion operation on the target storage volume based on the target long-term capacity demand trend prediction value, preset capacity expansion policy parameters, and target short-term capacity demand trend.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor is configured to execute the computer program to implement the steps of the method for elastically expanding a storage volume as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the storage volume elastic expansion method according to any one of claims 1 to 7.

Images