Dynamic resource allocation method and system oriented to heterogeneous storage

Abstract

The invention discloses a heterogeneous storage-oriented dynamic resource allocation method and system, and the method comprises the steps: carrying out the feature engineering of to-be-stored data and key feature data of each type of storage media, and obtaining input features; the feature engineering comprises key feature data preprocessing and feature construction according to the key feature data; then inputting the input features into an XGBoost model for prediction to obtain the probability of storing the to-be-stored data into each type of storage medium in the heterogeneous storage system; and finally, selecting the storage medium with the highest probability to store the to-be-stored data. According to the method, the problems of the heterogeneous storage system in performance, cost and dynamic workload adaptability are solved, and flexible and efficient storage resource allocation is realized.

Description

Technical Field

[0001] The present invention relates to dynamic resource allocation, and in particular to a dynamic resource allocation method and system for heterogeneous storage. Background Art

[0002] Against the backdrop of rapid development in fields such as big data, cloud computing, and high-performance computing, storage media, as critical infrastructure supporting data-intensive applications, are facing the dual challenges of performance and cost. Different storage media vary significantly in terms of cost, performance, and energy consumption. For example, traditional hard disk drives (HDDs) are widely used for data storage due to their low cost and large capacity, but their low throughput and high access latency limit their applicability in high-performance scenarios. Solid-state drives (SSDs) have become an important choice for performance-sensitive applications due to their significantly improved read and write speeds. New non-volatile memory (NVM) provides groundbreaking technical support for real-time data processing and high-frequency transactions with its extremely low latency and high throughput.

[0003] The existing storage resource allocation strategies mainly include hot and cold separation and cache design.

[0004] The hot-cold separation strategy divides data into "hot data" and "cold data" based on access frequency and optimizes performance and cost by using different storage media. Hot data refers to frequently accessed data that requires a fast response, while cold data refers to data that is rarely accessed or does not need to be accessed for a long time. For hot data, devices with high read / write speeds but high costs are selected as storage media, while cold data is stored on devices with lower read / write speeds but lower costs. However, the hot-cold separation strategy is limited in that it relies on static access frequency classification and may not be able to adapt to dynamic changes in data access patterns, resulting in inflexible resource allocation. For example, some data originally considered cold may suddenly become hot, and vice versa. In addition, the prediction of hot and cold data often relies on historical access data, which may not accurately predict future access patterns.

[0005] The cache design strategy can better meet the needs of different data access patterns by using different levels of storage media in combination. For example, for a heterogeneous storage system that uses three types of storage media: NVM, SSD, and HDD, NVM can be used as the cache layer of the storage system to store the most frequently accessed data. This data is usually the part that is accessed the most and requires an immediate response; SSD, as an intermediate layer storage, can be used to store data that is accessed moderately frequently but not frequently enough to be cached in NVM; HDD, as back-end storage, is used to store data that is not accessed for a long time. However, the limitation of the cache design strategy is that heterogeneous storage systems usually have multiple different levels of storage media. The design and management of each level needs to be optimized separately for business needs and workloads. This hierarchical design will limit the scalability of the entire system.

[0006] In summary, existing resource allocation methods cannot achieve flexible and efficient storage. Summary of the Invention

[0007] Purpose of the invention: The purpose of the present invention is to provide a dynamic resource allocation method and system for heterogeneous storage, which realizes intelligent resource allocation based on factors such as data access frequency, storage resource utilization and cost, combined with the performance requirements of real-time workloads, especially when the performance and cost characteristics of different storage media vary significantly, to achieve efficient storage resource allocation and scheduling, and solve the problem that traditional storage resource allocation strategies are difficult to flexibly adjust resources according to real-time business needs and dynamic load changes.

[0008] Technical solution: The dynamic resource allocation method for heterogeneous storage described in the present invention includes the following steps:

[0009] (1) performing feature engineering on the key feature data to be stored and each type of storage medium to obtain input features; the feature engineering includes preprocessing the key feature data and constructing features based on the key feature data;

[0010] (2) inputting the input features into the XGBoost model for prediction to obtain the probability of storing the data in each type of storage medium in the heterogeneous storage system;

[0011] (3) Select the storage medium with the highest probability to store the data to be stored.

[0012] Furthermore, the key feature data in step (1) includes the heat of the data to be stored:

[0013]

[0014] Among them, H i Indicates the heat of data i, F i,jrepresents the number of accesses to data i at time point j, α is the time decay coefficient, 0<α<1, t j represents the time difference between the current time and time point j;

[0015] The key feature data also includes the storage resource utilization of each storage medium:

[0016] U total =ω space ·U space +ω IOPS ·U IOPS ;

[0017] Among them U space 、U IOPS Represent storage space utilization and IOPS utilization, ω space 、ω IOPS Represents the weights of storage space utilization and IOPS utilization respectively;

[0018]

[0019] S used Indicates the storage space occupied by the storage medium, S total Indicates the total available space of the storage medium, IO current Indicates the actual number of I / O operations on the current storage medium, IO max Indicates the maximum I / O operation capability of the storage medium;

[0020] The key characteristic data also includes the performance and cost ratio of the storage medium:

[0021]

[0022] Where pf represents the hardware performance index of the storage medium, C unit Indicates the cost per unit storage capacity of a storage medium.

[0023] Furthermore, the feature construction in step (1) includes: constructing a storage pressure according to the access frequency of the data to be stored and the storage resource utilization of each type of storage medium, wherein the storage pressure is the ratio of the access frequency to the storage resource utilization.

[0024] Furthermore, the feature construction in step (1) includes: according to the performance-cost ratio PCR of the storage medium and the storage resource utilization U of the storage medium total Construct a cost-effective

[0025] Furthermore, the feature engineering in step (1) is iteratively optimized using a k-fold cross-validation method according to the training process and prediction process of the XGBoost model.

[0026] Furthermore, step (2) also includes calculating the SHAP value of each input feature to evaluate its importance;

[0027] Select input features with SHAP values higher than a threshold to input into the XGBoost model; or repeatedly delete input features with SHAP values and retrain the XGBoost model until the performance or cost is optimal, and then select the remaining input features to input into the XGBoost model.

[0028] Furthermore, step (2) also includes using the TreeSHAP approximation algorithm to calculate the SHAP value of each input feature for importance evaluation;

[0029] Select input features with SHAP values higher than a threshold to input into the XGBoost model; or repeatedly delete input features with SHAP values and retrain the XGBoost model until the performance or cost is optimal, and then select the remaining input features to input into the XGBoost model.

[0030] Furthermore, the method further includes step (4), monitoring the performance, load and resource usage of various storage media in real time, and returning to step (1).

[0031] The dynamic resource allocation system for heterogeneous storage described in the present invention includes:

[0032] A feature engineering module is used to perform feature engineering on the key feature data to be stored and each type of storage medium to obtain input features; the feature engineering includes preprocessing the key feature data and constructing features based on the key feature data;

[0033] A prediction module is used to input the input features into the XGBoost model for prediction, and obtain the probability of storing the data to be stored in each type of storage medium in the heterogeneous storage system;

[0034] The resource allocation module is used to select the storage medium with the highest probability to store the data to be stored.

[0035] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the dynamic resource allocation method for heterogeneous storage is implemented.

[0036] Beneficial effects: Compared with the existing technology, the advantages of the present invention are: (1) The present invention achieves a balance between performance and cost, and considers the performance and cost of the storage medium simultaneously in the dynamic resource allocation process. By constructing features (such as "price-performance ratio" features, "storage pressure" features, etc.), it ensures that while meeting the task performance requirements, the storage cost is reduced as much as possible, thereby achieving a dynamic balance between performance and cost. (2) The present invention evaluates the relevance and importance of each input feature through adaptive feature selection, adaptively selects the features most relevant to the workload requirements, and dynamically adjusts the feature set, which can automatically optimize the storage resource allocation strategy under different load scenarios. (3) The present invention has a real-time resource monitoring and feedback mechanism, which can adjust the storage strategy based on real-time data of load and resources, thereby coping with constantly changing workloads and improving the efficiency of resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the dynamic resource allocation method of the present invention.

[0038] Figure 2 This is a feature engineering flowchart of an embodiment of the present invention.

[0039] Figure 3 This is a workflow diagram of the XGBoost model according to an embodiment of the present invention.

[0040] Figure 4 This is a flowchart of the adaptive feature selection according to an embodiment of the present invention.

[0041] Figure 5 This is a flow chart of real-time resource monitoring and feedback according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] A heterogeneous storage system is a storage architecture that integrates at least two types of storage media. It achieves an optimal balance between performance and cost through dynamic resource allocation strategies combined with machine learning algorithms. This system automatically selects the optimal storage medium based on factors such as data access patterns and economic costs, leveraging the advantages of different storage media. The technical solutions of the present invention are further described below with reference to the accompanying figures.

[0043] like Figure 1 As shown, the dynamic resource allocation method for heterogeneous storage includes the following steps.

[0044] Step 1, data collection.

[0045] When optimizing data storage strategies, you first need to collect a series of key characteristic data in order to accurately evaluate and select the best storage media.

[0046] (1.1) Access frequency

[0047] Access frequency refers to how often a data block is accessed within a specific time window, typically measured as the number of accesses per unit time or data popularity. Data blocks with high access frequencies are typically hot data, requiring fast access. Therefore, these data are best stored on fast-response storage media (such as NVM or SSDs). Data blocks with low access frequencies are likely cold data, which can be stored on lower-cost but slower media like HDDs. By accurately capturing data block access patterns, the system can dynamically adjust storage policies to reduce wasted storage resources.

[0048] In order to quantify the access frequency of data blocks, the present invention uniformly defines the data block size as 4KB (i.e., 4096 bytes), and then uses the following formula for calculation:

[0049]

[0050] Among them: F i Indicates the access frequency of data block i, N i represents the number of accesses to data block i within a specific time window, and T represents the duration of the time window.

[0051] In addition, in order to adjust the storage strategy more flexibly, the present invention introduces a time decay factor based on the access frequency to capture the dynamic characteristics of the data block access frequency changing over time:

[0052]

[0053] Among them: H i Indicates the heat of data block i, F i,j represents the number of accesses to data block i at time point j, α represents the time decay coefficient, and its value range is 0<α<1. The closer to 1, the more emphasis is placed on recent accesses. j Represents the time difference between the current time and time point j.

[0054] (1.2) Storage resource utilization

[0055] Storage resource utilization reflects the current usage of various storage media (such as NVM, SSD, HDD), including indicators such as occupied space and IOPS (input and output operations per second). High storage utilization may mean that the storage medium is heavily loaded and needs to be optimized to avoid performance bottlenecks. Low storage utilization indicates that storage resources are idle and that existing resources can be used more efficiently by dynamically allocating data storage strategies. The present invention combines storage space utilization and IOPS utilization to more scientifically measure storage resource utilization:

[0056] U total =ω space ·U space+ω IOPS ·U IOPS ;

[0057] Among them: U total Indicates the comprehensive storage utilization, U space 、U IOPS Represent storage space utilization and IOPS utilization, ω space 、ω IOPS Represents the weights of storage space utilization and IOPS utilization respectively.

[0058] Storage space utilization: This metric measures the space occupied by storage media.

[0059]

[0060] Among them: U space Indicates storage space utilization, S used Indicates the storage space occupied by the storage medium, S total Indicates the total available space on the storage medium.

[0061] IOPS utilization measures the I / O operation load of the storage medium:

[0062]

[0063] Among them: U IOPS Indicates IOPS utilization, IO current Indicates the actual number of I / O operations of the current storage medium (unit: per second), IO max Indicates the maximum I / O operation capacity of the storage medium (unit: per second).

[0064] (1.3) Storage resource cost

[0065] The cost of storage media is also a factor that cannot be ignored in the decision-making process, and prices vary significantly among different storage media. NVM offers extremely fast read and write speeds, but comes at a high cost. SSDs offer a compromise between price and performance, while HDDs provide large storage capacity at a lower cost. By rationally weighing storage costs based on the specific storage requirements, an optimal balance between performance and cost can be achieved, thereby improving the economic efficiency of the system. To quantify storage costs and assist in the decision-making process for optimizing storage resources, this paper uses the following formula for measurement.

[0066] The cost per unit storage capacity:

[0067]

[0068] Where: C unitIndicates the cost per unit storage capacity (unit: yuan / GB), C total Indicates the total cost of storage media (unit: yuan), S total Indicates the total capacity of the storage medium (unit: GB).

[0069] In addition, in order to more broadly measure the performance and cost of storage resources, the present invention defines the following indicators:

[0070]

[0071] Among them: PCR (Performance-to-Cost Ratio) represents the ratio of performance to cost, pf represents the performance index of hardware (such as bandwidth, etc.), C unit Indicates the cost per unit storage capacity (unit: yuan / GB).

[0072] In an actual production environment, the weights of indicators such as capacity, bandwidth, unit storage capacity cost, and PCR can be flexibly adjusted according to demand.

[0073] Step 2, feature engineering.

[0074] After collecting the above data, it is necessary to convert the raw data into a format suitable for machine learning model input and extract important features that can help the model make better predictions. This process is called "feature engineering". Figure 2 As shown, feature engineering includes the following.

[0075] (1) Data preprocessing

[0076] Data preprocessing is the first step in feature engineering, which aims to clean and standardize the raw data.

[0077] First, missing values need to be processed and filled, as the raw data may contain missing values for various reasons. Features or samples with a large number of missing values should be directly deleted. For access frequency, the mean value of the specific storage medium (such as NVM, SSD, HDD) is used for filling. For storage resource utilization, the median or the value of adjacent time windows is used to maintain data smoothness. For storage resource cost, the average cost value of similar devices is used for filling, depending on the current storage medium type.

[0078] For discrete features, such as the type of storage medium (NVM, SSD, HDD) or system status (such as high load, low load), one-hot encoding is performed, such as NVM is represented as [1,0,0], SSD is represented as [0,1,0], HDD is represented as [0,0,1], etc.

[0079] Normalize or standardize continuous features to ensure that they are trained at the same scale to prevent certain features from affecting the model's training effect due to scale differences.

[0080] For access frequency and storage resource utilization, the present invention performs normalization:

[0081]

[0082] Where x' represents the normalized value, x represents the original data value, and x min Represents the minimum value of the feature in the entire data set, x max Represents the maximum value of a feature in the entire dataset.

[0083] For storage resource costs, the present invention performs standardization:

[0084]

[0085] Where x' represents the standardized value, x represents the original data value, μ represents the mean of the feature (the average value over the entire data set), and σ represents the standard deviation of the feature (reflecting the fluctuation range of the feature value).

[0086] (2) Feature construction

[0087] In some cases, the original features themselves may not be sufficient to provide sufficient information. In this case, you can consider constructing new features by combining or synthesizing the original features.

[0088] For example, access frequency and storage resource utilization can be combined into a new feature, "storage pressure", which may better reflect the intensity of data storage requirements:

[0089] P = F·U;

[0090] Here, F represents the access frequency and U represents the storage resource utilization.

[0091] In addition, we can also combine information such as storage resource utilization and cost to construct a "price / performance ratio" feature to measure the optimal choice of different storage media in the current environment:

[0092]

[0093] PCR represents the ratio of performance to cost, and U represents the storage resource utilization.

[0094] (3) Iterative Optimization

[0095] Feature engineering isn't a one-time effort; it's an ongoing, iterative process. As models are trained and feedback is received from real-world applications, you might find that certain features don't significantly improve model performance, or that new features can further improve prediction accuracy. Therefore, feature engineering requires continuous optimization, incorporating new data and workload characteristics to continuously refine the feature selection, transformation, and construction processes.

[0096] To verify the effectiveness of feature construction, we use a k-fold cross-validation method. This involves dividing the dataset into k subsets, selecting one of these subsets as the validation set and the remaining k-1 subsets as the training set. This process is repeated k times, each time using a different subset as the validation set. The final model performance is the average of all k validation results.

[0097] Step 3: Build a machine learning model.

[0098] In this application of storage resource optimization, XGBoost is used as a classification model to predict the most suitable medium for storing data. XGBoost is an efficient implementation of the Gradient Boosting Decision Tree (GBDT) algorithm. It calculates the probability of each storage medium based on the input features and uses the category with the highest probability as the final prediction result. The output of these categories can help the system automatically select the most suitable medium for storing data, ensuring the optimal balance between performance and cost of the storage system.

[0099] Specifically, the XGBoost model distributes the input data block or workload to one of multiple storage media, and then calculates a probability value for each storage medium. This value represents the likelihood of selecting the storage medium under the current input features. Finally, the system selects the storage medium with the highest probability value as the result. Figure 3 As shown, the workflow of the XGBoost model is as follows.

[0100] (3.1) Input features

[0101] The model receives feature inputs representing data blocks or workloads, including basic features (access frequency, storage resource utilization, storage resource cost) and constructed features (features generated through feature engineering).

[0102] (3.2) Calculate the probability

[0103] XGBoost generates a series of decision trees based on the input features, and each tree performs classification and scoring on the storage medium selection. During the model training process, the Softmax multi-classification loss function is used to calculate the probability of selecting each storage medium category. For K types of storage media:

[0104]

[0105] Where: P(y=k|x) represents the predicted probability that the data block belongs to category k, z k represents the original output of the model for category k (i.e. logits), represents a normalization term that ensures that the sum of the probabilities of all categories is 1.

[0106] For example, for a heterogeneous storage system using NVM, SSD, and HDD media, the XGBoost model generates raw logits (such as [z NVM =2.0,z SSD =1.0,z HDD =0.1]), and then use the Softmax function to convert logits into probabilities:

[0107]

[0108] The same applies to other media.

[0109] (3.3) Output prediction results

[0110] Finally, the XGBoost model will output the probability of each storage medium category, for example: P(NVM) = 0.7, P(SSD) = 0.2, P(HDD) = 0.1.

[0111] The system selects the storage medium with the highest probability (here, NVM) as the final result. The model's predictions help the system dynamically select the optimal storage medium and guide data block allocation, ensuring the optimal balance between storage system performance and cost. Furthermore, this prediction model can easily adapt to more storage media types (assuming new media needs to be added).

[0112] Step 4: Adaptive feature selection.

[0113] In hardware resource load prediction and heterogeneous storage systems, the feature data types used as input to the model vary (such as access frequency, storage utilization, IOPS, and storage cost), but the importance of these features can vary across scenarios or time points. For example, under high system load, I / O latency may be more critical, while under low load, features like access frequency have a greater impact on system decisions. Traditional fixed feature sets struggle to cope with varying loads and scenarios, resulting in insufficient model generalization.

[0114] Therefore, the present invention adds an adaptive feature selection module. Through an automatic feature selection algorithm, the input feature set is dynamically adjusted according to the characteristics of the dataset, thereby improving the prediction accuracy and generalization ability of the model in diverse scenarios. This module mainly achieves the following two goals:

[0115] ① Feature importance evaluation: Calculate the importance score of each input feature in real time.

[0116] ② Feature screening: Filter features that have a great impact on the current scene based on importance scores and discard unimportant features.

[0117] For example, under high load conditions, the module finds through real-time calculations that features such as "access frequency" have a greater impact on load prediction, and increases their weight while eliminating features such as "storage cost" that are insensitive in the short term; while under low load conditions, the module detects a decrease in system access volume, and "storage cost" becomes important as an optimization indicator. The feature set reintroduces the "storage cost" feature to reduce overall costs. Figure 4 As shown in Figure 2, the steps of adaptive feature selection are as follows.

[0118] (4.1) Feature Importance Assessment

[0119] This paper is based on SHAP values (Shapley Additive Explanations) and quantifies the impact of each feature on the output by calculating the contribution of each feature to the model prediction:

[0120]

[0121] Where: Φ m is the SHAP value of feature m, that is, the contribution of the mth feature to the model prediction result, X is the set of all features, Y is the feature subset that does not contain feature m, P 选 (Y) represents the prediction result output by the model when the input feature set is Y, P 选 (Y∪{m}) represents the prediction result after adding feature m to the set Y.

[0122] For example, assume that the current given input feature set is access frequency F, storage resource utilization U, storage cost C, and:

[0123] When the model does not input the above three features, P(NVM)=0.5;

[0124] When adding the input feature storage cost C, P(NVM)=0.4;

[0125] When adding input feature storage resource utilization U and storage cost C, P(NVM) = 0.6;

[0126] The final prediction result is P(NVM)=0.7.

[0127] For storage cost C: Φ C =P NVM ({C})-P NVM= 0.4 - 0.5 = -0.1. The calculation results show that when the model does not input the above three features, the probability of selecting NVM is 0.5. After adding the storage cost, the probability decreases to 0.4. Therefore, the contribution of C to the selection of NVM is -0.1, which is due to the high cost of NVM.

[0128] For access frequency F: Φ F =P NVM ({F,U,C})-P VNM ({U, C}) = 0.7 - 0.6 = 0.1. The calculation results show that when access frequency is not considered, the probability of selecting NVM is 0.6. After taking access frequency into account, the probability increases to 0.7. Therefore, F contributes +0.1 to the selection of NVM, indicating that access frequency characteristics are closely related to workloads with high bandwidth requirements.

[0129] This step requires calculating the contribution value of each feature, that is, the SHAP value, to calculate the marginal contribution of the feature in all possible subsets to evaluate its importance. Although in theory it is necessary to traverse all feature subsets, many methods can be used to simplify feature selection in practical applications. For example, the TreeSHAP approximation algorithm can be used, which optimizes the calculation process for the XGBoost tree model by utilizing the structural characteristics of the model (such as the splitting path of the tree) to avoid exhausting all permutations and combinations. At the same time, since TreeSHAP is essentially a mathematical optimization method to skip redundant calculations, it can ensure that the results are consistent with the exhaustive method, and its accuracy is not affected when the SHAP calculation formula remains unchanged.

[0130] (4.2) Feature screening

[0131] After the feature importance scoring is completed, different filtering conditions can be set according to different workload scenarios, such as:

[0132] ① Set threshold: Select features with scores above a certain threshold, such as SHAP values greater than 0.05

[0133] ② Recursive elimination: repeatedly remove the lowest-scoring features and retrain the model until the performance or cost is optimal.

[0134] Step 5: Real-time resource monitoring and feedback.

[0135] like Figure 5 As shown, it continuously monitors the performance, load, and resource usage of the storage system and provides real-time feedback to dynamically adjust storage policies to ensure that the storage system can respond to environmental changes in a timely manner while operating efficiently and optimize the allocation and utilization of storage resources.

[0136] Specifically, the real-time resource monitoring and feedback module continuously tracks key storage system metrics, such as the storage media's I / O performance (IOPS, throughput, and latency), CPU and memory resource utilization, and available storage capacity. These metrics reflect the current load on the storage system and help the system identify potential bottlenecks or resource shortages. For example, if a storage medium is overloaded, causing increased I / O operation latency, the system can adjust the storage decision module's strategy to select a less-loaded storage medium to alleviate the pressure and avoid performance degradation.

[0137] The monitoring module also collects dynamic operational information from the system, including the health status, faults, and anomalies of each storage medium. If a problem is detected, such as a hard drive failure or degraded storage medium performance, the monitoring module promptly notifies the system to make adjustments and reselect storage media, ensuring data security and system stability.

[0138] The feedback mechanism ensures continuous system optimization based on real-time data. When demand changes or load fluctuates, the real-time resource monitoring and feedback module provides the necessary information to dynamically adjust storage policies, ensuring efficient use of storage resources and avoiding resource waste, ultimately improving the overall performance and stability of the storage system.

[0139] The method of the present invention is introduced below through specific experiments.

[0140] A data center has deployed a heterogeneous storage system based on NVM, SSD, and HDD storage media. The system configuration is as follows:

[0141] Table 1 Introduction to heterogeneous storage systems

[0142]

[0143]

[0144] The PCR (bandwidth) of the three storage media in Table 1 are 0.73, 4.02 and 1.43 respectively.

[0145] The present invention can handle two different types of tasks:

[0146] High-load tasks: require fast response, data needs to be stored in high-speed storage media, and 200GB of storage capacity is required;

[0147] Low-load tasks: The data volume is large, but the performance requirements are relatively low. The data is stored in low-cost storage media and requires 1TB of storage capacity.

[0148] For ease of understanding, assume that the bandwidth requirement of low-load tasks is always 0.1GB / s, while the bandwidth requirement of high-load tasks is 8GB / s. Over time, the bandwidth requirement changes from 1GB / s to 8GB / s. Compare the cases without and with the present invention (excluding costs such as data migration):

[0149] If a static hot-cold separation strategy is adopted, all 200GB of data is stored in NVM, and 1TB of data is stored in HDD. The total storage resource cost is 2377.36 yuan.

[0150] Using the method of the present invention, the XGBoost machine learning model predicts and classifies data in high-load tasks. A quarter of the data read and write bandwidth requirements match NVM (3-8 GB / s), while three-quarters match SSD (1-3 GB / s). Therefore, the system retains 50 GB of data in NVM and then migrates 150 GB of data to SSD. The total storage resource cost is now 824.86 yuan. Using the present invention, the total storage resource cost is expected to be reduced by 1552.50 yuan.

[0151] Through the above analysis, it can be found that the present invention achieves more accurate resource scheduling by intelligently optimizing the resource allocation of heterogeneous storage, using machine learning models to predict and dynamically adjust the storage distribution of data, and ultimately reduces the total cost of storage resources.

[0152] The dynamic resource allocation system for heterogeneous storage described in the present invention includes:

[0153] A feature engineering module is used to perform feature engineering on the key feature data to be stored and each type of storage medium to obtain input features; the feature engineering includes preprocessing the key feature data and constructing features based on the key feature data;

[0154] A prediction module is used to input the input features into the XGBoost model for prediction, and obtain the probability of storing the data to be stored in each type of storage medium in the heterogeneous storage system;

[0155] The resource allocation module is used to select the storage medium with the highest probability to store the data to be stored.

[0156] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the dynamic resource allocation method for heterogeneous storage is implemented.

Claims

1. A dynamic resource allocation method for heterogeneous storage, characterized in that: The steps include: (1) performing feature engineering on the key feature data to be stored and each type of storage medium to obtain input features; the feature engineering includes preprocessing the key feature data and constructing features based on the key feature data; (2) inputting the input features into the XGBoost model for prediction to obtain the probability of storing the data in each type of storage medium in the heterogeneous storage system; (3) Select the storage medium with the highest probability to store the data to be stored.

2. The dynamic resource allocation method for heterogeneous storage according to claim 1, characterized in that: The key feature data in step (1) includes the heat of the data to be stored: Among them, H i Indicates the heat of data i, F i,j represents the number of accesses to data i at time point j, α is the time decay coefficient, 0<α<1, t j represents the time difference between the current time and time point j; The key feature data also includes the storage resource utilization of each storage medium: IN total =ω space ·IN space +ω IOPS ·IN IOPS ; Among them U space 、U IOPS Represent storage space utilization and IOPS utilization, ω space 、ω IOPS Represents the weights of storage space utilization and IOPS utilization respectively; S used Indicates the storage space occupied by the storage medium, S total Indicates the total available space of the storage medium, IO current Indicates the actual number of I / O operations on the current storage medium, IO max Indicates the maximum I / O operation capability of the storage medium; The key characteristic data also includes the performance and cost ratio of the storage medium: Where pf represents the hardware performance index of the storage medium, C unit Indicates the cost per unit storage capacity of a storage medium.

3. The dynamic resource allocation method for heterogeneous storage according to claim 2, characterized in that: The feature construction in step (1) includes: constructing storage pressure according to the access frequency of the data to be stored and the storage resource utilization of each type of storage medium, and the storage pressure is the ratio of the access frequency to the storage resource utilization.

4. The dynamic resource allocation method for heterogeneous storage according to claim 2, characterized in that: The feature construction in step (1) includes: according to the performance and cost ratio PCR of the storage medium and the storage resource utilization rate U of the storage medium total Construct a cost-effective 5. The dynamic resource allocation method for heterogeneous storage according to claim 1, characterized in that: The feature engineering in step (1) is iteratively optimized using the k-fold cross-validation method according to the training process and prediction process of the XGBoost model.

6. The dynamic resource allocation method for heterogeneous storage according to claim 1, characterized in that: Step (2) also includes calculating the SHAP value of each input feature for importance assessment; Select input features with SHAP values higher than a threshold to input into the XGBoost model; or repeatedly delete input features with SHAP values and retrain the XGBoost model until the performance or cost is optimal, and then select the remaining input features to input into the XGBoost model.

7. The dynamic resource allocation method for heterogeneous storage according to claim 1, characterized in that: Step (2) also includes using the TreeSHAP approximation algorithm to calculate the SHAP value of each input feature for importance assessment; Select input features with SHAP values higher than a threshold to input into the XGBoost model; or repeatedly delete input features with SHAP values and retrain the XGBoost model until the performance or cost is optimal, and then select the remaining input features to input into the XGBoost model.

8. The dynamic resource allocation method for heterogeneous storage according to claim 1, characterized in that: The method further includes step (4), monitoring the performance, load and resource usage of various storage media in real time, and returning to step (1).

9. A dynamic resource allocation system for heterogeneous storage based on the method of claim 1, characterized in that: include: A feature engineering module is used to perform feature engineering on the key feature data to be stored and each type of storage medium to obtain input features; the feature engineering includes preprocessing the key feature data and constructing features based on the key feature data; A prediction module is used to input the input features into the XGBoost model for prediction, and obtain the probability of storing the data to be stored in each type of storage medium in the heterogeneous storage system; The resource allocation module is used to select the storage medium with the highest probability to store the data to be stored.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into a processor, the dynamic resource allocation method for heterogeneous storage according to any one of claims 1 to 8 is implemented.

Images