Storage resource allocation and capacity prediction method and system for large-scale data center

Abstract

The invention provides a storage resource allocation and capacity prediction method and system for a large-scale data center, and relates to the technical field of data processing, and the method comprises the steps: obtaining the historical resource use data of a plurality of storage nodes, constructing a time sequence prediction model based on a multi-level feature extraction network, dividing the storage nodes into a plurality of node groups with different load levels, a differential evolution algorithm is adopted to optimize prediction parameters, a hierarchical resource allocation scheme is generated, resources are allocated according to priorities, and an elastic expansion threshold value is set, so that the accuracy and the dynamic property of resource allocation are realized, and the utilization rate of storage resources and the overall operation efficiency of the data center are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a storage resource allocation and capacity prediction method and system for a large-scale data center. Background Art

[0002] With the rapid development of cloud computing and big data technologies, large-scale data centers have become critical infrastructure supporting a wide range of internet services. These data centers typically contain thousands of storage nodes, generating and processing massive amounts of data daily. Rationally allocating and predicting data center storage resource requirements is crucial for ensuring service quality, reducing operating costs, and improving resource utilization.

[0003] Traditional data center storage resource management primarily relies on static allocation or dynamic adjustment based on simple thresholds. Static allocation often reserves resources based on peak demand, resulting in significant idle resources. Simple threshold adjustment methods, on the other hand, are unable to cope with complex and volatile business load patterns, often resulting in untimely or over-allocation of resources. As business scale and complexity increase, these traditional methods are no longer able to meet the demands for efficient operations in large-scale data centers.

[0004] Existing technologies have the following shortcomings in storage resource allocation and capacity prediction in large-scale data centers: First, existing prediction methods generally lack the ability to fine-tune modeling of different load characteristics and are unable to make differentiated predictions for the load characteristics of different types of storage nodes, resulting in insufficient prediction accuracy; second, most resource allocation strategies use a unified standard to process all storage nodes and fail to implement hierarchical management based on node importance and load characteristics, resulting in a situation where critical and non-critical businesses compete for resources. Summary of the Invention

[0005] The embodiments of the present invention provide a storage resource allocation and capacity prediction method and system for a large-scale data center, which can solve the problems in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a method for intelligently allocating storage resources and predicting capacity in a large-scale data center, comprising: Obtaining historical resource usage data of multiple storage nodes in the large-scale data center; Based on the historical resource usage data, a multi-level feature extraction network is used to extract features and build a time series prediction model; Dividing the plurality of storage nodes into a plurality of node groups with different load levels, optimizing the prediction parameters of the time series prediction model using a differential evolution algorithm, and generating corresponding prediction models for the node groups with different load levels; Using the prediction models corresponding to the node groups with different load levels, storage capacity demand prediction is performed respectively to obtain prediction results for each node group; Using a combinatorial optimization algorithm to generate a hierarchical resource allocation plan based on the prediction results of each node group, and setting resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; Executing the hierarchical resource allocation scheme to allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priorities; The actual storage capacity usage of the storage nodes is monitored, and in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold, additional storage resources are allocated from a resource pool of the large-scale data center.

[0007] Based on the historical resource usage data, a multi-level feature extraction network is used to extract features and build a time series prediction model, including: Performing time series segmentation based on the historical resource usage data to generate a training data sequence; Constructing a multi-level feature extraction network, configuring a plurality of parallel feature extraction channels in the multi-level feature extraction network, and using the plurality of parallel feature extraction channels to perform feature extraction on the training data sequence to obtain a plurality of groups of time series features; A self-attention mechanism module is provided in the multi-level feature extraction network, wherein the self-attention mechanism module calculates a feature weight matrix for the multiple groups of temporal features; Performing weighted fusion on the multiple groups of time series features according to the feature weight matrix to generate a fused feature vector; Inputting the fused feature vector into a long short-term memory network for training to obtain an initial prediction model; Using a validation data set to validate the initial prediction model to obtain a prediction error; When the prediction error exceeds a preset threshold, the number of the multiple parallel feature extraction channels and the weight parameters of the self-attention mechanism module are adjusted, and the feature extraction, weighted fusion and model training of the training data sequence are re-performed until the prediction error is less than the preset threshold, thereby obtaining the time series prediction model.

[0008] Dividing the plurality of storage nodes into a plurality of node groups with different load levels, optimizing prediction parameters of the time series prediction model using a differential evolution algorithm, and generating corresponding prediction models for the node groups with different load levels, including: Acquire historical load data of multiple storage nodes, perform normalization processing on the historical load data, and generate a standardized load feature matrix; Building a load scoring model based on the standardized load feature matrix, and calculating load scoring values ​​of the plurality of storage nodes using the load scoring model; Dividing the plurality of storage nodes into node groups with different load levels according to the load score values; An adaptive parameter optimization module is constructed using a differential evolution algorithm. The adaptive parameter optimization module generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels. Building a prediction model for the node groups of different load levels based on the prediction model parameters and the standardized load characteristic matrix; The prediction accuracy of the prediction model is verified using the test data in the standardized load characteristic matrix. In response to the prediction accuracy failing to meet the preset requirements, the evolutionary generations and mutation probability in the adaptive parameter optimization module are adjusted, and the parameter optimization is re-executed using the standardized load characteristic matrix until a prediction model that meets the preset requirements is generated.

[0009] An adaptive parameter optimization module is constructed using a differential evolution algorithm. The adaptive parameter optimization module generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels, including: Determining the encoding length of the parameter to be optimized according to the standardized load characteristic matrix, generating a parameter encoding vector, setting the population size according to the parameter encoding vector, and generating an initial population matrix; Constructing a multi-layer neural network based on the standardized load feature matrix, extracting the time series correlation features and periodic fluctuation features of node groups with different load levels, and combining the time series correlation features and the periodic fluctuation features to generate a dynamic weight vector; Using the dynamic weight vector to construct an adaptive mutation factor for the individuals in the initial population matrix, the adaptive mutation factor automatically adjusts the variable step length according to the changing intensity of the time series correlation feature; A memory fitting module is constructed based on the periodic fluctuation characteristics, wherein the memory fitting module stores the optimal solution evolution path in the historical optimization process and uses the optimal solution evolution path as a constraint condition to guide the dynamic adjustment of the crossover probability; Performing a mutation operation using the adaptive mutation factor to generate a mutation vector; Determining a crossover probability according to the memory fitting module, and performing a crossover operation on the mutation vector; Select and update the population matrix and decode to obtain the prediction model parameters corresponding to the node groups with different load levels.

[0010] The prediction models corresponding to the node groups with different load levels are used to respectively predict the storage capacity demand, and the prediction results of each node group are obtained, including: Obtaining capacity usage data streams of node groups with different load levels, performing data cleaning based on the capacity usage data streams, and generating normalized time series data; Constructing a dual-attention-enhanced hybrid prediction model, wherein the dual-attention-enhanced hybrid prediction model includes a temporal feature attention module and a node group association attention module; The temporal feature attention module calculates the feature weights of key time points in the predicted input sequence, and the node group association attention module calculates the load migration association strength between node groups of different load levels; constructing a combined prediction function based on the feature weights and the load migration correlation strength, and applying the combined prediction function to the normalized time series data; Using the combined prediction function to perform capacity prediction on each load level node group to generate capacity prediction data; A confidence interval analysis is performed on the capacity prediction data. In response to a confidence interval exceeding a preset range, the characteristic weight and the strength of association with the load migration are adjusted, and the prediction calculation is re-executed until a prediction result for each node group is obtained.

[0011] A combinatorial optimization algorithm is used to generate a hierarchical resource allocation plan based on the prediction results of each node group, and resource allocation priorities and elastic scaling thresholds are set for the node groups of different load levels, including: Obtain the forecast results of each node group, extract the time series fluctuation characteristics of the forecast data, and generate a multi-dimensional resource demand matrix; Building a node group resource affinity model based on the multi-dimensional resource demand matrix, calculating the load migration probability between node groups, and obtaining priority constraints; Establishing a resource elasticity coefficient model based on the multi-dimensional resource demand matrix to obtain a dynamic threshold value of system resources; Constructing a combined optimization model based on demand forecasting, wherein the combined optimization model maps the multidimensional resource demand matrix into an objective function, maps the priority constraint condition and the system resource dynamic threshold into a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem and obtain optimization parameters; constructing a hierarchical resource allocation scheme using the optimization parameters in combination with the load migration probability; Extracting a node group priority sequence based on the hierarchical resource allocation scheme and the resource elasticity coefficient model to obtain resource allocation priorities corresponding to node groups of different load levels; The elastic scaling thresholds corresponding to node groups of different load levels are determined according to the node group priority sequence and the system resource dynamic threshold.

[0012] A combined optimization model based on demand forecasting is constructed. The combined optimization model maps the multidimensional resource demand matrix into an objective function, maps the priority constraint condition and the system resource dynamic threshold into a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem, thereby obtaining optimization parameters, including: Performing feature decomposition on the multi-dimensional resource demand matrix to obtain a temporal feature matrix and a spatial feature matrix; Constructing an objective function based on the temporal feature matrix and the spatial feature matrix, and constructing a constraint function based on the priority constraint condition and the system resource dynamic threshold; Constructing a hierarchical competitiveness evaluation module to calculate the demand competition intensity of different resource dimensions based on the temporal feature matrix and the spatial feature matrix to obtain an inter-layer competitiveness coefficient; Performing selective hierarchical mapping on the temporal feature matrix and the spatial feature matrix using the inter-layer competition coefficient to generate a feature mapping matrix; Constructing an adaptive weight matrix based on the feature mapping matrix, wherein the adaptive weight matrix is ​​used to dynamically adjust the weight ratio of the objective function to the constraint function; Performing a weighted combination of the objective function and the constraint function using the adaptive weight matrix to obtain a comprehensive optimization function; The comprehensive optimization function is solved to obtain a candidate solution set, the candidate solution set is screened according to the constraint function to obtain a solution vector that satisfies the constraint, and the solution vector is converted into an optimization parameter.

[0013] A second aspect of an embodiment of the present invention provides a system for intelligently allocating storage resources and predicting capacity for a large-scale data center, comprising: The first unit is configured to obtain historical resource usage data of multiple storage nodes in the large-scale data center; The second unit is configured to extract features based on the historical resource usage data using a multi-level feature extraction network to construct a time series prediction model; A third unit is configured to divide the plurality of storage nodes into a plurality of node groups with different load levels, optimize the prediction parameters of the time series prediction model using a differential evolution algorithm, and generate corresponding prediction models for the node groups with different load levels; A fourth unit is configured to respectively perform storage capacity demand forecasting on the node groups with different load levels using the forecasting models corresponding to the node groups with different load levels, and obtain a forecasting result for each node group; A fifth unit is configured to generate a hierarchical resource allocation plan based on the prediction results of each node group using a combinatorial optimization algorithm, and set resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; A sixth unit is configured to execute the hierarchical resource allocation scheme and allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priority; The seventh unit is configured to monitor actual storage capacity usage of the storage nodes, and allocate additional storage resources from a resource pool of the large-scale data center in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold.

[0014] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0015] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0016] The beneficial effects of this application are as follows: By acquiring historical resource usage data of storage nodes and building a time series prediction model based on a multi-level feature extraction network, the present invention can accurately capture the complex patterns of data center resource usage, improve prediction accuracy, and make resource allocation more reasonable and efficient.

[0017] The present invention groups storage nodes according to load levels and uses a differential evolution algorithm to optimize prediction parameters, achieving accurate prediction and customized resource management for different types of nodes, avoiding the "one-size-fits-all" resource allocation strategy in traditional methods, and significantly improving storage resource utilization.

[0018] The present invention introduces hierarchical resource allocation and dynamic elastic scaling mechanisms, which can reasonably allocate low-load node resources while ensuring the service quality of high-load nodes. The system can automatically trigger resource expansion based on real-time monitoring data, effectively preventing the risk of storage capacity exhaustion and improving the operation and maintenance efficiency and reliability of the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for allocating storage resources and predicting capacity in a large-scale data center according to an embodiment of the present invention; Figure 2 This is a schematic diagram showing the performance comparison of storage node load prediction based on the differential evolution algorithm; Figure 3 Schematic diagram of the simulation results of the adaptive parameter optimization module of the differential evolution algorithm. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 FIG. 1 is a flow chart of a method for allocating storage resources and predicting capacity of a large-scale data center according to an embodiment of the present invention. Figure 1 As shown, the method includes: Obtaining historical resource usage data of multiple storage nodes in the large-scale data center; Based on the historical resource usage data, a multi-level feature extraction network is used to extract features and build a time series prediction model; Dividing the plurality of storage nodes into a plurality of node groups with different load levels, optimizing the prediction parameters of the time series prediction model using a differential evolution algorithm, and generating corresponding prediction models for the node groups with different load levels; Using the prediction models corresponding to the node groups with different load levels, storage capacity demand prediction is performed respectively to obtain prediction results for each node group; Using a combinatorial optimization algorithm to generate a hierarchical resource allocation plan based on the prediction results of each node group, and setting resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; Executing the hierarchical resource allocation scheme to allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priorities; The actual storage capacity usage of the storage nodes is monitored, and in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold, additional storage resources are allocated from a resource pool of the large-scale data center.

[0023] In an optional embodiment, based on the historical resource usage data, a multi-level feature extraction network is used to extract features and construct a time series prediction model, including: Performing time series segmentation based on the historical resource usage data to generate a training data sequence; Constructing a multi-level feature extraction network, configuring a plurality of parallel feature extraction channels in the multi-level feature extraction network, and using the plurality of parallel feature extraction channels to perform feature extraction on the training data sequence to obtain a plurality of groups of time series features; A self-attention mechanism module is provided in the multi-level feature extraction network, wherein the self-attention mechanism module calculates a feature weight matrix for the multiple groups of temporal features; Performing weighted fusion on the multiple groups of time series features according to the feature weight matrix to generate a fused feature vector; Inputting the fused feature vector into a long short-term memory network for training to obtain an initial prediction model; Using a validation data set to validate the initial prediction model to obtain a prediction error; When the prediction error exceeds a preset threshold, the number of the multiple parallel feature extraction channels and the weight parameters of the self-attention mechanism module are adjusted, and the feature extraction, weighted fusion and model training of the training data sequence are re-performed until the prediction error is less than the preset threshold, thereby obtaining the time series prediction model.

[0024] This embodiment first obtains historical resource usage data, which includes time series data such as server CPU usage, memory usage, and disk IO. The obtained historical resource usage data is segmented into time series, and the resource usage data of consecutive time periods is divided into multiple training samples in a sliding window manner. Specifically, a window size of 60 minutes and a step size of 5 minutes are selected, that is, every 60 minutes of data is used as a sample, and each time it slides back 5 minutes, the next sample is generated. For example, if the historical resource usage data is a record of the CPU usage of a server for 7 consecutive days and the sampling frequency is once per minute, approximately 2016 training samples can be generated.

[0025] A multi-level feature extraction network is then constructed, with three parallel feature extraction channels configured within the network: a time-domain feature extraction channel, a frequency-domain feature extraction channel, and a statistical feature extraction channel. The time-domain feature extraction channel uses a one-dimensional convolutional layer with eight convolution kernels, a kernel size of 3, and a stride of 1 to extract local pattern features from time series data. The frequency-domain feature extraction channel first performs a Fourier transform on the input data, converting it to the frequency domain. Frequency-domain features are then extracted through a one-dimensional convolutional layer with six convolution kernels, a kernel size of 5, and a stride of 1. The statistical feature extraction channel calculates statistics of the input data, including mean, variance, kurtosis, and skewness, and maps them into high-dimensional feature representations through a fully connected layer.

[0026] For the time-domain feature extraction channel, the input is a 60-dimensional vector (corresponding to 60 minutes of data). After processing through a one-dimensional convolutional layer, a 58×8-dimensional feature map is obtained, which is then processed through a maximum pooling layer to obtain a 29×8-dimensional feature map. For the frequency-domain feature extraction channel, the input data undergoes a Fourier transform to obtain a frequency-domain representation. A one-dimensional convolutional layer extracts 56×6-dimensional features, which are then processed through maximum pooling to obtain a 28×6-dimensional feature map. For the statistical feature extraction channel, 10 statistics are calculated and mapped to a 32-dimensional feature vector through a fully connected layer.

[0027] A self-attention mechanism module is set up in the multi-level feature extraction network to perform weighted fusion of the features extracted from the three channels. The self-attention module receives the above three sets of features as input and generates a feature weight matrix. In specific implementation, the three sets of features are first flattened and concatenated to obtain a comprehensive feature vector with a size of 29×8+28×6+32=412 dimensions. Then, through a two-layer fully connected network, the features are mapped to a 3-dimensional weight vector, representing the importance weights of the three feature channels. In practical applications, for CPU usage prediction tasks, the weight of the time domain feature channel is usually about 0.45, the weight of the frequency domain feature channel is about 0.35, and the weight of the statistical feature channel is about 0.2. The specific weight values ​​will be automatically adjusted and optimized during subsequent training.

[0028] The three sets of time series features are weighted and fused according to the feature weight matrix to generate a fused feature vector. Specifically, the three sets of features are multiplied by their corresponding weights and then concatenated into a fused feature vector. For example, if the weight of the time domain features is 0.45, the weight of the frequency domain features is 0.35, and the weight of the statistical features is 0.2, then the dimension of the fused feature vector is 29 × 8 × 0.45 + 28 × 6 × 0.35 + 32 × 0.2 = 232 dimensions.

[0029] The fused feature vector is input into a long short-term memory (LSTM) network for training to obtain an initial prediction model. The LSTM network consists of two LSTM layers, with 64 neurons in the first layer and 32 neurons in the second layer. Finally, a fully connected layer outputs the prediction results. Training uses the Adam optimizer with a learning rate of 0.001, a batch size of 32, and 100 training epochs. The root mean square error (RMSE) is used as the loss function to measure the difference between the predicted value and the true value.

[0030] The initial prediction model was validated using a validation dataset to obtain the prediction error. The validation dataset consisted of 20% of the historical resource usage data, which was not used in the training process. During validation, the average prediction error of the model on the validation dataset was calculated, such as the root mean square error between the predicted CPU usage and the actual CPU usage. The preset threshold was set at 3%, meaning that the average error between the predicted and actual values ​​did not exceed 3 percentage points.

[0031] When the prediction error exceeds a preset threshold, it is necessary to adjust the number of parallel feature extraction channels and the weight parameters of the self-attention mechanism module. For example, if the initial model prediction error is 4.5%, exceeding the preset 3% threshold, the following adjustments can be made: increase the number of convolution kernels in the time domain feature extraction channel from 8 to 12; adjust the convolution kernel size in the frequency domain feature extraction channel from 5 to 3; add more statistical features to the statistical feature channel, such as quantiles and peak-to-valley ratios; and adjust the number of neurons in the fully connected layer of the self-attention mechanism to make the weight calculation more accurate.

[0032] After the adjustments, the training data sequence is re-processed for feature extraction, weighted fusion, and model training. In practice, 3-5 iterations of adjustments are typically required until the prediction error falls below a preset threshold. For example, after three iterations of adjustments, the prediction error drops from an initial 4.5% to 2.8%, below the preset 3% threshold. This is when the final time series prediction model is obtained.

[0033] The resulting prediction model can predict resource usage for the next 15 minutes based on 60 minutes of historical resource usage data, with an accuracy rate exceeding 97%. This model can be used for data center resource scheduling, foreseeing potential resource shortages in advance, and enabling appropriate task allocation and resource expansion.

[0034] In an optional embodiment, the plurality of storage nodes are divided into a plurality of node groups with different load levels, a differential evolution algorithm is used to optimize the prediction parameters of the time series prediction model, and corresponding prediction models are generated for the node groups with different load levels, including: Acquire historical load data of multiple storage nodes, perform normalization processing on the historical load data, and generate a standardized load feature matrix; Building a load scoring model based on the standardized load feature matrix, and calculating load scoring values ​​of the plurality of storage nodes using the load scoring model; Dividing the plurality of storage nodes into node groups with different load levels according to the load score values; An adaptive parameter optimization module is constructed using a differential evolution algorithm. The adaptive parameter optimization module generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels. Building a prediction model for the node groups of different load levels based on the prediction model parameters and the standardized load characteristic matrix; The prediction accuracy of the prediction model is verified using the test data in the standardized load characteristic matrix. In response to the prediction accuracy failing to meet the preset requirements, the evolutionary generations and mutation probability in the adaptive parameter optimization module are adjusted, and the parameter optimization is re-executed using the standardized load characteristic matrix until a prediction model that meets the preset requirements is generated.

[0035] The present invention discloses a method for dividing multiple storage nodes into node groups with different load levels and optimizing the prediction parameters of a time series prediction model using a differential evolution algorithm. The specific implementation is as follows: In actual applications, the system first obtains historical load data for multiple storage nodes, including metrics such as CPU usage, memory utilization, network transmission rate, and disk I / O. The system normalizes this historical load data, converting load metrics of different dimensions to the [0, 1] range to eliminate dimensionality effects. For example, the CPU usage data for a storage node over the past 30 days, {78%, 85%, 92%, 64%, 79%}, is normalized to {0.5, 0.75, 1.0, 0.0, 0.54}. By performing the same operation on the multi-dimensional load metrics of all storage nodes, a standardized load feature matrix is ​​generated.

[0036] Based on the standardized load characteristic matrix, the system constructs a load scoring model. This model comprehensively considers the importance weights of various load indicators to calculate a comprehensive load score. Assuming the system configuration has a CPU utilization weight of 0.4, a memory usage weight of 0.3, a network transmission rate weight of 0.2, and a disk I / O weight of 0.1, for storage nodes with standardized load indicator values ​​of {0.75, 0.6, 0.8, 0.9}, the load score is calculated as 0.75 × 0.4 + 0.6 × 0.3 + 0.8 × 0.2 + 0.9 × 0.1 = 0.72. The system performs the same calculation for all storage nodes to obtain a load score for each node.

[0037] Based on the calculated load scores, the system divides storage nodes into groups with different load levels. For example, nodes with load scores in the range [0, 0.3) are classified as the low-load group, nodes in the range [0.3, 0.7) are classified as the medium-load group, and nodes in the range [0.7, 1.0] are classified as the high-load group. In practice, after load score calculation, 50 storage nodes in a data center were divided into 12 low-load nodes, 28 medium-load nodes, and 10 high-load nodes.

[0038] The system utilizes a differential evolution algorithm to construct an adaptive parameter optimization module. First, an initial population solution is generated based on a standardized load characteristic matrix. For each node group with a load level, the system randomly generates an initial population of 30 individuals, each of which contains the parameter set required for the prediction model. For example, for a time series prediction model, parameters might include the time window size, the number of hidden layer nodes, and the learning rate. The value of each parameter in the initial population is randomly generated within a preset range. For example, the time window size is randomly selected from the range [5, 20], the number of hidden layer nodes is randomly selected from the range [10, 100], and the learning rate is randomly selected from the range [0.001, 0.1].

[0039] The generated initial population solution is optimized through crossover and mutation operations. The system sets the crossover probability to 0.8, the mutation probability to 0.5, and the maximum number of evolutionary generations to 100. In each generation of evolution, the system performs a differential mutation operation on the individuals in the population. This involves randomly selecting three different individuals and generating mutant individuals through vector differentials. A crossover operation is then performed to exchange some parameters of the mutant individuals with those of the original individuals to generate test individuals. Finally, a selection operation is performed to compare the prediction accuracy of the test individuals with that of the original individuals on the training dataset, retaining the individuals with higher accuracy for the next generation. Through multiple generations of evolution, the system generates optimized prediction model parameters for each load level node group.

[0040] Based on the optimized prediction model parameters and the standardized load characteristic matrix, the system constructs prediction models for node groups with different load levels. For example, the optimal parameters for the high-load node group are {time window = 12, number of hidden layer nodes = 64, learning rate = 0.05}, and the system constructs the corresponding prediction model based on this configuration. The optimal parameters for the medium-load node group are {time window = 8, number of hidden layer nodes = 32, learning rate = 0.03}, and the optimal parameters for the low-load node group are {time window = 6, number of hidden layer nodes = 16, learning rate = 0.01}.

[0041] The system verifies the prediction model's accuracy using test data from the standardized load signature matrix. If the prediction model for a load-level node group achieves an average absolute percentage error of 8% on the test data, exceeding the preset requirement of 5%, the system adjusts the number of evolutionary generations and mutation probability in the adaptive parameter optimization module, for example, increasing the maximum number of evolutionary generations to 200 and the mutation probability to 0.6. The system then re-executes the parameter optimization process using the standardized load signature matrix. After this new round of optimization, the average absolute percentage error of the prediction model drops to 4.2%, meeting the preset requirement. The system then saves the prediction model for deployment.

[0042] In practice, after deploying this method in a cloud storage system, load prediction accuracy improved by 25% for high-load node groups, 18% for medium-load node groups, and 12% for low-load node groups. Overall system resource utilization increased by 15%, task response time decreased by 20%, and service quality significantly improved. A differentiated prediction model based on load levels can more accurately capture the changing patterns of nodes with different load characteristics, providing a reliable basis for resource scheduling and load balancing.

[0043] Figure 2 This is a schematic diagram of the performance comparison of storage node load prediction based on differential evolution algorithm. Figure 2 As shown in the figure, the system performance improvement under different load level node groups and different task types is shown; it can be seen from the figure that the method of the present invention has obvious improvements in the three key indicators of prediction accuracy, resource utilization and response time, among which the performance improvement of high-load node groups and computing-intensive tasks is the most significant, reaching 25.0%, 22.0% and 27.0% respectively.

[0044] In an optional embodiment, a differential evolution algorithm is used to construct an adaptive parameter optimization module, which generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels, including: Determining the encoding length of the parameter to be optimized according to the standardized load characteristic matrix, generating a parameter encoding vector, setting the population size according to the parameter encoding vector, and generating an initial population matrix; Constructing a multi-layer neural network based on the standardized load feature matrix, extracting the time series correlation features and periodic fluctuation features of node groups with different load levels, and combining the time series correlation features and the periodic fluctuation features to generate a dynamic weight vector; Using the dynamic weight vector to construct an adaptive mutation factor for the individuals in the initial population matrix, the adaptive mutation factor automatically adjusts the variable step length according to the changing intensity of the time series correlation feature; A memory fitting module is constructed based on the periodic fluctuation characteristics, wherein the memory fitting module stores the optimal solution evolution path in the historical optimization process and uses the optimal solution evolution path as a constraint condition to guide the dynamic adjustment of the crossover probability; Performing a mutation operation using the adaptive mutation factor to generate a mutation vector; Determining a crossover probability according to the memory fitting module, and performing a crossover operation on the mutation vector; Select and update the population matrix and decode to obtain the prediction model parameters corresponding to the node groups with different load levels.

[0045] In this embodiment, in order to realize the construction of an adaptive parameter optimization module using the differential evolution algorithm, the system first determines the encoding length of the parameters to be optimized based on the standardized load feature matrix. For node groups with different load levels, the parameter encoding vector contains several prediction model parameters. For example, for low-load node groups, the encoding length is 8, including parameters such as the time window size and the number of hidden layer neurons; for medium-load node groups, the encoding length is 12, and parameters such as feature extraction depth are added; for high-load node groups, the encoding length is 16, and parameters such as the attention mechanism weight are further added. The population size is set to 50, and an initial population matrix of 50× encoding length is generated. Each element in the matrix is ​​randomly initialized in the range [0,1].

[0046] The system constructs a three-layer neural network to extract features based on the standardized load feature matrix. The number of nodes in the input layer is consistent with the number of load features, the number of nodes in the hidden layer is twice that of the input layer, and the output layer contains two nodes, corresponding to the time series correlation feature and the periodic fluctuation feature, respectively. The time series correlation feature characterizes the intensity of load changes and is obtained by calculating the difference sequence of load data at adjacent time points. The numerical range is between [-1, 1]. The periodic fluctuation feature characterizes the periodicity of load changes and is obtained by extracting the main frequency components through Fourier analysis. The numerical range is between [0, 1]. These two types of features are combined to generate a dynamic weight vector. The dimension of the weight vector is consistent with the parameter encoding vector, and the weight of each dimension is set according to the sensitivity of the corresponding parameter to the two types of features.

[0047] An adaptive mutation factor is constructed for the individuals in the initial population matrix using a dynamic weight vector. The mutation factor F consists of a base mutation factor F0 and an adaptive adjustment term ΔF. F0 is set to 0.5, and ΔF is dynamically adjusted based on the intensity of the time series correlation feature changes. The greater the intensity of the change, the larger ΔF, enabling the algorithm to perform large-step exploration even when the load fluctuates significantly. For example, when the time series correlation eigenvalue is 0.8, ΔF is 0.3, resulting in a final mutation factor F of 0.8; when the eigenvalue is 0.2, ΔF is 0.1, resulting in a final F of 0.6.

[0048] The system constructs a memory fitting module based on cyclical fluctuation characteristics. This module maintains a historical queue of 10 optimized solutions, recording the evolution path of the optimal solution during the historical optimization process. The memory fitting module selects the appropriate historical optimization path as a constraint based on the similarity between the current cyclical fluctuation characteristics and historical data. For example, when the current cyclical fluctuation characteristic value is 0.75, which has the highest similarity to the historical data value of 0.72, the system extracts the optimization trajectory under this condition and calculates the boundaries of the optimal solution region ([0.3, 0.7], [0.2, 0.5], etc.), which serve as the basis for adjusting the crossover probability.

[0049] The system performs a mutation operation using an adaptive mutation factor. Three different individual vectors are randomly selected from the initial population and their weighted differences are added to the target vector to generate the mutation vector. For the i-th individual in the population matrix, the three randomly selected vectors have indices r1=12, r2=35, and r3=47. Based on the previously calculated mutation factor F=0.8, a differential mutation operation is performed on the vectors to generate the mutation vector.

[0050] The system determines the crossover probability (CR) based on the memory fitting module. The base crossover probability (CR0) is set to 0.5, and the adjustment term (ΔCR) is calculated based on the constraints of the historical optimization path. When the current solution is near the historical optimal solution region, reducing the crossover probability promotes local search; when the current solution is far from the historical optimal solution region, increasing the crossover probability promotes global exploration. For example, if the normalized distance of the current solution from the historical optimal solution region is 0.4, the ΔCR is calculated to be 0.2, and the final crossover probability (CR) is 0.7. Based on this crossover probability, the system performs a crossover operation on the mutation vector to generate a test vector.

[0051] The system selects and updates the population matrix by comparing the objective function values. The objective function is defined as the mean squared error of the prediction model on the validation set. For each individual, if the objective function value of its test vector is better than that of the original individual, the test vector replaces the original individual; otherwise, the original individual is retained. After 100 generations of iterative optimization, the optimal individual in the population matrix is ​​decoded to obtain the prediction model parameters corresponding to node groups with different load levels. For example, for the low-load node group, the decoded parameters include a time window size of 8 and a number of hidden layer neurons of 32; for the medium-load node group, the decoded parameters include a time window size of 12, a number of hidden layer neurons of 64, and a feature extraction depth of 3; for the high-load node group, the decoded parameters include a time window size of 16, a number of hidden layer neurons of 128, a feature extraction depth of 4, and attention mechanism weights of [0.4, 0.3, 0.2, 0.1].

[0052] Figure 3 This is a schematic diagram of the simulation results of the adaptive parameter optimization module of the differential evolution algorithm. Through experimental analysis, it can be found that the adaptive differential evolution algorithm improves the optimization convergence speed by 28.7% compared with the standard differential evolution algorithm by dynamically adjusting the mutation factor and crossover probability, and reduces the parameter optimization time by 32.6%.

[0053] Using time-series correlation features to adjust the variation factor can effectively adapt to the intensity of load changes. Large-step exploration is used when the load fluctuates violently, while small-step fine search is used when the load is stable. The memory fitting module constructed based on periodic fluctuation characteristics can effectively utilize historical optimization experience and achieve a balance between global exploration and local fine search by dynamically adjusting the crossover probability. The optimization effect of the prediction model parameters for node groups with different load levels is significant, especially the MSE of the high-load node group is reduced by 54.3%, which proves the adaptability of this method to complex load scenarios. The prediction model optimized by the adaptive differential evolution algorithm also shows significant improvement in prediction time. The prediction time of the low, medium and high load groups is reduced by 27.1%, 23.4% and 18.2% respectively, which improves the overall response efficiency of the system.

[0054] In an optional embodiment, the prediction models corresponding to the node groups with different load levels are used to respectively predict storage capacity demand, and the prediction results of each node group are obtained, including: Obtaining capacity usage data streams of node groups with different load levels, performing data cleaning based on the capacity usage data streams, and generating normalized time series data; Constructing a dual-attention-enhanced hybrid prediction model, wherein the dual-attention-enhanced hybrid prediction model includes a temporal feature attention module and a node group association attention module; The temporal feature attention module calculates the feature weights of key time points in the predicted input sequence, and the node group association attention module calculates the load migration association strength between node groups of different load levels; constructing a combined prediction function based on the feature weights and the load migration correlation strength, and applying the combined prediction function to the normalized time series data; Using the combined prediction function to perform capacity prediction on each load level node group to generate capacity prediction data; A confidence interval analysis is performed on the capacity prediction data. In response to a confidence interval exceeding a preset range, the characteristic weight and the strength of association with the load migration are adjusted, and the prediction calculation is re-executed until a prediction result for each node group is obtained.

[0055] Exemplarily, prediction models corresponding to node groups with different load levels are used to predict storage capacity demand, generating prediction results for each node group. The method first obtains capacity usage data streams for node groups with different load levels. These data streams contain the capacity usage of each node group in the real-time monitored storage system, such as hourly recorded disk usage, data write volume, and data read volume. For the raw data stream, the system performs a data cleansing process, including outlier detection and processing, missing value filling, and noise smoothing. For example, if the storage utilization of a node group suddenly jumps from 50% to 99% at a specific time point, the system marks these anomalous data points and replaces them with the average of the nearby valid data points. Missing data caused by monitoring interruptions is filled using linear interpolation. Random fluctuations in the data are smoothed using a moving average filter, with a smoothing window size set to 6 hours to preserve the seasonal variation characteristics of the data. The cleaned data is then normalized, mapping each indicator value to a range of 0 to 1 to generate normalized time series data.

[0056] Constructing a hybrid prediction model enhanced with dual attention is a core component of the prediction process. This model comprises a temporal feature attention module and an inter-node group association attention module. The temporal feature attention module calculates the feature weights of key time points in the prediction input sequence. In specific implementation, the system uses the past 30 days of historical data as the input sequence, sets the initial weights of the data at each time point in the sequence to equal values, and then calculates the importance score of each time point through a multi-layer perceptron network. For example, during peak business hours such as 9:00-11:00 and 14:00-16:00 daily, the system may assign higher weights, such as 0.15, to these time points, while assigning lower weights, such as 0.05, to low-traffic hours at night, such as 2:00-5:00 a.m. The temporal feature attention mechanism enables the model to automatically learn and focus on historical data points with predictive value.

[0057] The node group correlation attention module calculates the load migration correlation strength between node groups of different load levels. In an actual data center environment, there is a load migration phenomenon between node groups of different load levels. For example, when a high-load node group (utilization rate > 85%) approaches its capacity limit, new data may be allocated to a medium-load node group (utilization rate 50%-85%). The system establishes a correlation matrix by analyzing the correlation between the capacity changes of different node groups in the past 7 days. For example, the correlation strength between the high-load node group and the medium-load node group may be 0.7, indicating that there is a strong load migration relationship between the two; while the correlation strength between the high-load and low-load node groups may be only 0.3, indicating that the possibility of direct migration is low.

[0058] Based on the correlation between feature weights and load migration, the system constructs a combined prediction function. This function integrates the prediction results of a long short-term memory network (for processing time series features) and a graph convolutional network (for processing relationships between node groups). The weight coefficient of the combined prediction function is determined by both the time series feature attention and the node group association attention. For example, when predicting a high-load node group, when the historical data for that group is relatively stable, the time series feature weight may be set to 0.7 and the association weight to 0.3. However, when a clear load migration trend is observed, the time series feature weight may be adjusted to 0.4 and the association weight increased to 0.6. The system applies the combined prediction function to the normalized time series data to predict the storage capacity requirements of each load level node group for the next seven days.

[0059] After using a combined prediction function to perform capacity forecasts for each load-level node group, the system generates capacity forecast data. For a high-load node group, for example, the forecast indicates that its storage utilization rate will gradually increase from the current 87% to 93% in seven days. To assess the reliability of the forecast results, the system performs confidence interval analysis on the capacity forecast data and calculates a 95% confidence interval. If the confidence interval of the forecast result is 93% ± 5%, which exceeds the preset ± 3% range, the system automatically adjusts the model parameters. This adjustment process involves reallocating the weights of time series features, such as increasing the weight of the last three days' data to 0.6 and decreasing the weight of earlier data to 0.4. It also optimizes the correlation strength between node groups, such as increasing the correlation coefficient with adjacent load-level node groups. The system reruns the forecast calculation until the confidence interval is narrowed to within the preset range, ultimately outputting the storage capacity demand forecast for each node group.

[0060] Table 1 is a comparison table of key performance indicators of the system:

[0061] As shown in Table 1, it is clear that the dual-attention-enhanced hybrid prediction model significantly outperforms traditional prediction methods in all key performance indicators, especially in emergency response capabilities and prediction execution speed. This shows that the model not only improves prediction accuracy, but also greatly improves the system's response speed and adaptability.

[0062] Through the above method, data center managers can accurately predict the changing trends of storage capacity requirements of node groups with different load levels, plan capacity expansion in a timely manner, avoid the risk of business interruption caused by storage resource exhaustion, and optimize resource allocation to improve the overall efficiency and reliability of the storage system.

[0063] In an optional embodiment, a combinatorial optimization algorithm is used to generate a hierarchical resource allocation scheme based on the prediction results of each node group, and resource allocation priorities and elastic scaling thresholds are set for the node groups of different load levels, including: Obtain the forecast results of each node group, extract the time series fluctuation characteristics of the forecast data, and generate a multi-dimensional resource demand matrix; Building a node group resource affinity model based on the multi-dimensional resource demand matrix, calculating the load migration probability between node groups, and obtaining priority constraints; Establishing a resource elasticity coefficient model based on the multi-dimensional resource demand matrix to obtain a dynamic threshold value of system resources; Constructing a combined optimization model based on demand forecasting, wherein the combined optimization model maps the multidimensional resource demand matrix into an objective function, maps the priority constraint condition and the system resource dynamic threshold into a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem and obtain optimization parameters; constructing a hierarchical resource allocation scheme using the optimization parameters in combination with the load migration probability; Extracting a node group priority sequence based on the hierarchical resource allocation scheme and the resource elasticity coefficient model to obtain resource allocation priorities corresponding to node groups of different load levels; The elastic scaling thresholds corresponding to node groups of different load levels are determined according to the node group priority sequence and the system resource dynamic threshold.

[0064] In practice, the system first obtains the forecast results for each node group, extracts the time-series fluctuation characteristics of the forecast data, and generates a multidimensional resource demand matrix. For example, assuming there are five node groups, each with a forecast for CPU, memory, network bandwidth, and storage requirements for the next 24 hours, the system constructs a 5×24×4 three-dimensional matrix. For node group A at time t, the CPU utilization may be 80%, the memory utilization 65%, the network bandwidth utilization 45%, and the storage utilization 55%. When extracting fluctuation characteristics, the standard deviation, peak factor, and coefficient of variation of each resource in the time dimension are calculated. For example, the standard deviation of CPU utilization for node group A is 10%, the peak factor is 1.5, and the coefficient of variation is 0.12. These characteristic values ​​constitute the multidimensional resource demand matrix, which serves as the foundational data for subsequent optimization.

[0065] A node group resource affinity model is constructed based on the multi-dimensional resource demand matrix, and the probability of load migration between node groups is calculated to obtain priority constraints. The resource affinity model evaluates the feasibility of load migration by calculating the similarity of resource usage patterns between node groups. For example, for node groups A and B, the system calculates the correlation coefficient of their usage patterns in each resource dimension. Assume that the correlation coefficient of the CPU usage patterns of A and B is 0.85, indicating a high correlation; while the correlation coefficient of the memory usage patterns is only 0.3, indicating a low correlation. Taking into account the correlation of each resource dimension, the system calculates that the probability of load migration from A to B is 0.65, and the probability of migration from B to A is 0.45. These migration probabilities constitute the priority constraints for resource allocation, ensuring the rational allocation of resources between node groups.

[0066] A resource elasticity coefficient model is established based on the multidimensional resource demand matrix to obtain dynamic thresholds for system resources. The resource elasticity coefficient model quantifies the sensitivity of a node group to resource changes. For example, by analyzing the historical data of node group A, the system found that when CPU resources increased by 10%, performance improved by 15%, and the CPU elasticity coefficient was calculated to be 1.5; however, a 10% increase in memory resources only resulted in a 5% performance improvement, and the memory elasticity coefficient was calculated to be 0.5. The system sets dynamic thresholds based on the resource elasticity coefficient of each node group and the overall resource status of the system. For example, for CPU resources, the system may set a lower threshold of 40% and an upper threshold of 90%; for memory resources, the lower threshold is 30% and the upper threshold is 85%. These thresholds are dynamically adjusted as the system load changes.

[0067] A combinatorial optimization model based on demand forecasting is constructed. This model maps the multidimensional resource demand matrix into an objective function, and maps priority constraints and dynamic thresholds of system resources into constraint functions. The objective function aims to maximize resource utilization while minimizing resource conflicts. The constraint function ensures that priority relationships and resource constraints between node groups are met. The system uses a weight allocation strategy that is adaptive to resource contention to solve the problem. Specifically, when the contention level of a certain resource type (such as the CPU) is detected to be higher than a preset threshold (such as 85%), the system increases the weight of that resource in the objective function. For example, the weight of the CPU resource is adjusted from an initial 0.3 to 0.5, while the weights of other resources are reduced accordingly. Through an iterative optimization algorithm, the system ultimately obtains the optimized parameters, including the amount of resources that should be allocated to each node group at different points in time.

[0068] A hierarchical resource allocation scheme is constructed using optimized parameters combined with load migration probability. This scheme specifies the resource quotas and adjustment strategies that each node group should receive under different load conditions. For example, for node group A, when the load is low (CPU utilization <40%), the base resource quota (4-core CPU, 8GB of memory) is allocated; when the load is medium (CPU utilization 40%-70%), the resource quota is increased by 25% (5-core CPU, 10GB of memory); and when the load is high (CPU utilization >70%), the resource quota is increased by 50% (6-core CPU, 12GB of memory). The allocation scheme also considers the possibility of load migration between node groups, reserving the necessary resource buffer.

[0069] Based on the hierarchical resource allocation scheme and resource elasticity coefficient model, the node group priority sequence is extracted to obtain the resource allocation priority corresponding to node groups with different load levels. The system calculates the priority score based on the business importance of the node group, the resource elasticity coefficient, and the load forecast results. For example, the business importance coefficient of node group A is 0.9, the average resource elasticity coefficient is 1.2, and the load forecast shows that it will enter a peak period. The final calculated priority score is 8.5; the business importance coefficient of node group B is 0.7, the average resource elasticity coefficient is 0.9, the load forecast is stable, and the priority score is 6.2. Node groups are sorted from high to low according to the priority score to form a priority sequence, such as A>C>E>B>D.

[0070] The elastic scaling thresholds for node groups at different load levels are determined based on the node group priority sequence and the dynamic thresholds for system resources. High-priority node groups receive more relaxed scaling thresholds, while low-priority node groups receive stricter ones. For example, for the highest-priority node group A, the expansion threshold for CPU utilization is set at 70%, and the reduction threshold is set at 30%. For the lower-priority node group D, the expansion threshold is set at 85%, and the reduction threshold is set at 40%. This setting ensures that high-priority businesses receive resource guarantees in advance, while low-priority businesses prioritize resource efficiency. The system dynamically adjusts these thresholds regularly (e.g., hourly) based on actual load conditions and forecast results to adapt to business changes.

[0071] In an optional embodiment, a combined optimization model based on demand forecasting is constructed. The combined optimization model maps the multidimensional resource demand matrix as an objective function, maps the priority constraint condition and the system resource dynamic threshold as a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem to obtain optimization parameters, including: Performing feature decomposition on the multi-dimensional resource demand matrix to obtain a temporal feature matrix and a spatial feature matrix; Constructing an objective function based on the temporal feature matrix and the spatial feature matrix, and constructing a constraint function based on the priority constraint condition and the system resource dynamic threshold; Constructing a hierarchical competitiveness evaluation module to calculate the demand competition intensity of different resource dimensions based on the temporal feature matrix and the spatial feature matrix to obtain an inter-layer competitiveness coefficient; Performing selective hierarchical mapping on the temporal feature matrix and the spatial feature matrix using the inter-layer competition coefficient to generate a feature mapping matrix; Constructing an adaptive weight matrix based on the feature mapping matrix, wherein the adaptive weight matrix is ​​used to dynamically adjust the weight ratio of the objective function to the constraint function; Performing a weighted combination of the objective function and the constraint function using the adaptive weight matrix to obtain a comprehensive optimization function; The comprehensive optimization function is solved to obtain a candidate solution set, the candidate solution set is screened according to the constraint function to obtain a solution vector that satisfies the constraint, and the solution vector is converted into an optimization parameter.

[0072] In a specific implementation, a method for constructing a combined optimization model based on demand forecasting maps the multidimensional resource demand matrix into an objective function, and at the same time maps the priority constraints and the system resource dynamic threshold into a constraint function, and obtains the optimization parameters through a weight allocation strategy that is adaptive to resource competition.

[0073] The system first performs eigendecomposition on the multidimensional resource demand matrix to obtain a temporal feature matrix and a spatial feature matrix. Assume the multidimensional resource demand matrix is ​​a 10×8 matrix representing the demand values ​​for 8 different resource types at 10 time points. After eigendecomposition, a 10×4 temporal feature matrix and a 4×8 spatial feature matrix are obtained. The feature dimension is set to 4, determined by calculating the cumulative contribution rate of the eigenvalues ​​to be greater than 95%.

[0074] An objective function is constructed based on the temporal and spatial feature matrices. This objective function is designed to maximize the benefits of resource allocation while optimizing the balance of resource usage. For example, the system calculates the benefit coefficient for each resource dimension: 0.8 for CPU resources, 0.6 for memory resources, 0.5 for storage resources, and 0.7 for network bandwidth resources. These benefit coefficients are multiplied by the corresponding resource demands to form the benefit function. Balance is measured by calculating the variance of the utilization rates of different resource dimensions; smaller variances indicate more balanced resource usage.

[0075] A constraint function is constructed based on priority constraints and dynamic system resource thresholds. Priority constraints specify the conditions that must be met before higher-priority tasks, while dynamic system resource thresholds are dynamically adjusted based on system status. For example, consider a task pool containing five tasks with priorities of 1, 2, 3, 1, and 2 (lower values ​​indicate higher priorities). Dynamic system resource thresholds are set to: CPU utilization not exceeding 85%, memory utilization not exceeding 90%, storage utilization not exceeding 75%, and network bandwidth utilization not exceeding 80%.

[0076] A hierarchical competition assessment module was constructed. Based on the temporal and spatial feature matrices, the intensity of demand competition for different resource dimensions was calculated to obtain the inter-layer competition coefficient. This calculation method analyzes the demand fluctuations for each resource dimension at different time points. Greater fluctuations indicate more intense competition. The ratio of the standard deviation of demand to the mean is calculated as a fluctuation indicator. A larger ratio indicates higher competition. For example, the CPU resource contention is 0.35, the memory resource contention is 0.28, the storage resource contention is 0.15, and the network bandwidth resource contention is 0.42.

[0077] The temporal feature matrix and the spatial feature matrix are selectively mapped hierarchically using the inter-layer competition coefficient to generate a feature mapping matrix. The principle of hierarchical mapping is to assign a more detailed mapping structure to dimensions with high resource competition, while dimensions with low competition can adopt a coarser-grained mapping. For resource dimensions with competition greater than 0.3 (such as CPU and network bandwidth), a four-layer mapping structure is used; for resource dimensions with competition between 0.2 and 0.3 (such as memory), a three-layer mapping structure is used; for resource dimensions with competition less than 0.2 (such as storage), a two-layer mapping structure is used.

[0078] An adaptive weight matrix is ​​constructed based on the feature map matrix. This matrix is ​​used to dynamically adjust the weight ratio between the objective function and the constraint function. The construction of the weight matrix takes into account resource contention, priority, and historical allocation effects. The more competitive a resource dimension is, the greater its weight is during the optimization process. For example, the weight of network bandwidth resources is set to 0.32, the weight of CPU resources is set to 0.28, the weight of memory resources is set to 0.22, and the weight of storage resources is set to 0.18.

[0079] An adaptive weight matrix is ​​used to perform a weighted combination of the objective function and the constraint function to produce a comprehensive optimization function. This weighted combination uses a linear weighting scheme, namely, the weighted sum of the objective function and the constraint function. In practice, the objective function and constraint function weight ratio is set at 7:3, with the objective function weighting at 0.7 and the constraint function weighting at 0.3. As the system operates, if constraint violations increase, the ratio is dynamically adjusted to 6:4 or 5:5 to strengthen the influence of the constraints.

[0080] The comprehensive optimization function is solved to obtain a set of candidate solutions. Here, an improved particle swarm optimization algorithm is used. The number of particles is set to 50, the number of iterations is set to 100, and the convergence threshold is set to 0.001. After each iteration, the particle positions represent a set of resource allocation options, forming the candidate solution set. The candidate solution set is filtered based on the constraint function, removing solutions that violate the constraints and obtaining a solution vector that satisfies the constraints.

[0081] The process of converting the solution vector into optimization parameters includes calculating resource allocation ratios, determining the task scheduling order, and dividing resource allocation into time windows. For example, the solution vector might be represented as a 40-length vector, where the first 8 elements represent the allocation ratios of the eight types of resources in time period 1, and so on. The converted optimization parameters include: the allocation ratio matrix for each type of resource in each time period (a 10×8 matrix), the priority sequence for task execution (a 5-length vector representing the task ID), and the time window division for resource allocation (10 time points are divided into three time windows, each containing 3, 4, and 3 time points, respectively).

[0082] Through the above implementation method, the system can adaptively adjust the weights of optimization objectives and constraints based on the multi-dimensional resource demand prediction results, taking into account factors such as resource competition and task priority, and ultimately obtain resource optimization allocation parameters that meet system requirements, thereby maximizing resource utilization efficiency.

[0083] The storage resource intelligent allocation and capacity prediction system for a large-scale data center according to an embodiment of the present invention includes: The first unit is configured to obtain historical resource usage data of multiple storage nodes in the large-scale data center; The second unit is configured to extract features based on the historical resource usage data using a multi-level feature extraction network to construct a time series prediction model; A third unit is configured to divide the plurality of storage nodes into a plurality of node groups with different load levels, optimize the prediction parameters of the time series prediction model using a differential evolution algorithm, and generate corresponding prediction models for the node groups with different load levels; A fourth unit is configured to respectively perform storage capacity demand forecasting on the node groups with different load levels using the forecasting models corresponding to the node groups with different load levels, and obtain a forecasting result for each node group; A fifth unit is configured to generate a hierarchical resource allocation plan based on the prediction results of each node group using a combinatorial optimization algorithm, and set resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; A sixth unit is configured to execute the hierarchical resource allocation scheme and allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priority; The seventh unit is configured to monitor actual storage capacity usage of the storage nodes, and allocate additional storage resources from a resource pool of the large-scale data center in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold.

[0084] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0085] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0086] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A storage resource allocation and capacity prediction method for a large-scale data center, characterized in that: include: Obtaining historical resource usage data of multiple storage nodes in the large-scale data center; Based on the historical resource usage data, a multi-level feature extraction network is used to extract features and build a time series prediction model; Dividing the plurality of storage nodes into a plurality of node groups with different load levels, optimizing the prediction parameters of the time series prediction model using a differential evolution algorithm, and generating corresponding prediction models for the node groups with different load levels; Using the prediction models corresponding to the node groups with different load levels, storage capacity demand prediction is performed respectively to obtain prediction results for each node group; Using a combinatorial optimization algorithm to generate a hierarchical resource allocation plan based on the prediction results of each node group, and setting resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; Executing the hierarchical resource allocation scheme to allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priorities; The actual storage capacity usage of the storage nodes is monitored, and in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold, additional storage resources are allocated from a resource pool of the large-scale data center.

2. The method according to claim 1, characterized in that Based on the historical resource usage data, a multi-level feature extraction network is used to extract features and build a time series prediction model, including: Performing time series segmentation based on the historical resource usage data to generate a training data sequence; Constructing a multi-level feature extraction network, configuring a plurality of parallel feature extraction channels in the multi-level feature extraction network, and using the plurality of parallel feature extraction channels to perform feature extraction on the training data sequence to obtain a plurality of groups of time series features; A self-attention mechanism module is provided in the multi-level feature extraction network, wherein the self-attention mechanism module calculates a feature weight matrix for the multiple groups of temporal features; Performing weighted fusion on the multiple groups of time series features according to the feature weight matrix to generate a fused feature vector; Inputting the fused feature vector into a long short-term memory network for training to obtain an initial prediction model; Using a validation data set to validate the initial prediction model to obtain a prediction error; When the prediction error exceeds a preset threshold, the number of the multiple parallel feature extraction channels and the weight parameters of the self-attention mechanism module are adjusted, and the feature extraction, weighted fusion and model training of the training data sequence are re-performed until the prediction error is less than the preset threshold, thereby obtaining the time series prediction model.

3. The method according to claim 1, characterized in that Dividing the plurality of storage nodes into a plurality of node groups with different load levels, optimizing prediction parameters of the time series prediction model using a differential evolution algorithm, and generating corresponding prediction models for the node groups with different load levels, including: Acquire historical load data of multiple storage nodes, perform normalization processing on the historical load data, and generate a standardized load feature matrix; Building a load scoring model based on the standardized load feature matrix, and calculating load scoring values ​​of the plurality of storage nodes using the load scoring model; Dividing the plurality of storage nodes into node groups with different load levels according to the load score values; An adaptive parameter optimization module is constructed using a differential evolution algorithm. The adaptive parameter optimization module generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels. Building a prediction model for the node groups of different load levels based on the prediction model parameters and the standardized load characteristic matrix; The prediction accuracy of the prediction model is verified using the test data in the standardized load characteristic matrix. In response to the prediction accuracy failing to meet the preset requirements, the evolutionary generations and mutation probability in the adaptive parameter optimization module are adjusted, and the parameter optimization is re-executed using the standardized load characteristic matrix until a prediction model that meets the preset requirements is generated.

4. The method according to claim 3, characterized in that An adaptive parameter optimization module is constructed using a differential evolution algorithm. The adaptive parameter optimization module generates an initial population solution based on the standardized load characteristic matrix, optimizes the initial population solution through a crossover mutation operation, and generates corresponding prediction model parameters for the node groups of different load levels, including: Determining the encoding length of the parameter to be optimized according to the standardized load characteristic matrix, generating a parameter encoding vector, setting the population size according to the parameter encoding vector, and generating an initial population matrix; Constructing a multi-layer neural network based on the standardized load feature matrix, extracting the time series correlation features and periodic fluctuation features of node groups with different load levels, and combining the time series correlation features and the periodic fluctuation features to generate a dynamic weight vector; Using the dynamic weight vector to construct an adaptive mutation factor for the individuals in the initial population matrix, the adaptive mutation factor automatically adjusts the variable step length according to the changing intensity of the time series correlation feature; A memory fitting module is constructed based on the periodic fluctuation characteristics, wherein the memory fitting module stores the optimal solution evolution path in the historical optimization process and uses the optimal solution evolution path as a constraint condition to guide the dynamic adjustment of the crossover probability; Performing a mutation operation using the adaptive mutation factor to generate a mutation vector; Determining a crossover probability according to the memory fitting module, and performing a crossover operation on the mutation vector; Select and update the population matrix and decode to obtain the prediction model parameters corresponding to the node groups with different load levels.

5. The method according to claim 1, characterized in that The prediction models corresponding to the node groups with different load levels are used to respectively predict the storage capacity demand, and the prediction results of each node group are obtained, including: Obtaining capacity usage data streams of node groups with different load levels, performing data cleaning based on the capacity usage data streams, and generating normalized time series data; Constructing a dual-attention-enhanced hybrid prediction model, wherein the dual-attention-enhanced hybrid prediction model includes a temporal feature attention module and a node group association attention module; The temporal feature attention module calculates the feature weights of key time points in the predicted input sequence, and the node group association attention module calculates the load migration association strength between node groups of different load levels; constructing a combined prediction function based on the feature weights and the load migration correlation strength, and applying the combined prediction function to the normalized time series data; Using the combined prediction function to perform capacity prediction on each load level node group to generate capacity prediction data; A confidence interval analysis is performed on the capacity prediction data. In response to a confidence interval exceeding a preset range, the characteristic weight and the strength of association with the load migration are adjusted, and the prediction calculation is re-executed until a prediction result for each node group is obtained.

6. The method according to claim 1, characterized in that A combinatorial optimization algorithm is used to generate a hierarchical resource allocation plan based on the prediction results of each node group, and resource allocation priorities and elastic scaling thresholds are set for the node groups of different load levels, including: Obtain the forecast results of each node group, extract the time series fluctuation characteristics of the forecast data, and generate a multi-dimensional resource demand matrix; Building a node group resource affinity model based on the multi-dimensional resource demand matrix, calculating the load migration probability between node groups, and obtaining priority constraints; Establishing a resource elasticity coefficient model based on the multi-dimensional resource demand matrix to obtain a dynamic threshold value of system resources; Constructing a combined optimization model based on demand forecasting, wherein the combined optimization model maps the multidimensional resource demand matrix into an objective function, maps the priority constraint condition and the system resource dynamic threshold into a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem and obtain optimization parameters; constructing a hierarchical resource allocation scheme using the optimization parameters in combination with the load migration probability; Extracting a node group priority sequence based on the hierarchical resource allocation scheme and the resource elasticity coefficient model to obtain resource allocation priorities corresponding to node groups of different load levels; The elastic scaling thresholds corresponding to node groups of different load levels are determined according to the node group priority sequence and the system resource dynamic threshold.

7. The method according to claim 6, characterized in that A combined optimization model based on demand forecasting is constructed. The combined optimization model maps the multidimensional resource demand matrix into an objective function, maps the priority constraint condition and the system resource dynamic threshold into a constraint function, and adopts a resource contention adaptive weight allocation strategy to solve the problem, thereby obtaining optimization parameters, including: Performing feature decomposition on the multi-dimensional resource demand matrix to obtain a temporal feature matrix and a spatial feature matrix; Constructing an objective function based on the temporal feature matrix and the spatial feature matrix, and constructing a constraint function based on the priority constraint condition and the system resource dynamic threshold; Constructing a hierarchical competitiveness evaluation module to calculate the demand competition intensity of different resource dimensions based on the temporal feature matrix and the spatial feature matrix to obtain an inter-layer competitiveness coefficient; Performing selective hierarchical mapping on the temporal feature matrix and the spatial feature matrix using the inter-layer competition coefficient to generate a feature mapping matrix; Constructing an adaptive weight matrix based on the feature mapping matrix, wherein the adaptive weight matrix is ​​used to dynamically adjust the weight ratio of the objective function to the constraint function; Performing a weighted combination of the objective function and the constraint function using the adaptive weight matrix to obtain a comprehensive optimization function; The comprehensive optimization function is solved to obtain a candidate solution set, the candidate solution set is screened according to the constraint function to obtain a solution vector that satisfies the constraint, and the solution vector is converted into an optimization parameter.

8. A storage resource allocation and capacity prediction system for a large-scale data center, configured to implement the method according to any one of claims 1 to 7, characterized in that: include: The first unit is configured to obtain historical resource usage data of multiple storage nodes in the large-scale data center; The second unit is configured to extract features based on the historical resource usage data using a multi-level feature extraction network to construct a time series prediction model; A third unit is configured to divide the plurality of storage nodes into a plurality of node groups with different load levels, optimize the prediction parameters of the time series prediction model using a differential evolution algorithm, and generate corresponding prediction models for the node groups with different load levels; A fourth unit is configured to respectively perform storage capacity demand forecasting on the node groups with different load levels using the forecasting models corresponding to the node groups with different load levels, and obtain a forecasting result for each node group; A fifth unit is configured to generate a hierarchical resource allocation plan based on the prediction results of each node group using a combinatorial optimization algorithm, and set resource allocation priorities and elastic scaling thresholds for the node groups of different load levels; A sixth unit is configured to execute the hierarchical resource allocation scheme and allocate storage resources on the storage nodes of the node groups with different load levels according to the resource allocation priority; The seventh unit is configured to monitor actual storage capacity usage of the storage nodes, and allocate additional storage resources from a resource pool of the large-scale data center in response to a storage node whose actual storage capacity usage exceeds the elastic scaling threshold.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images