A node automatic scaling method and system of a distributed storage system

Abstract

The application relates to a node automatic scaling method and system of a distributed storage system, and the method comprises the following steps: obtaining network flow data and making the network flow data into a data set; building a deep learning network model for regression prediction; training the deep learning network model by using the data set to obtain a flow prediction model for network flow prediction; obtaining real-time network flow, obtaining future network flow by the flow prediction model, calculating storage data demand corresponding to the future network flow, calculating the demand node number of hadoop according to the storage data demand, if the current node number of hadoop is adapted to the demand node number, the node number of hadoop remains unchanged, otherwise, the current node number of hadoop is adjusted to adapt to the demand node number. Compared with the prior art, the application is based on hadoop for distributed storage and processing, the number of nodes is automatically scaled by learning and predicting the flow trend through a deep learning algorithm, the storage service quality is guaranteed, and the storage resources of the distributed storage system are effectively saved.

Description

Technical Field

[0001] This invention relates to the fields of data analysis and distributed storage, and in particular to a method and system for automatic scaling of nodes in a distributed storage system. Background Technology

[0002] The Hadoop Distributed File System (HDFS) is a core component of Hadoop, serving as an underlying distributed storage service. It addresses the problem of big data storage. HDFS consists of storage systems spanning multiple computers. It has broad application prospects in the era of big data, providing the necessary scalability for storing and processing massive amounts of data.

[0003] Hadoop cluster size and processing capacity are typically set based on peak demand for cluster resources, inevitably leading to resource waste during off-peak periods. While research on automatic scaling of distributed nodes exists, most current studies rely on experience and demand to manually set elastic scaling thresholds, failing to truly achieve dynamic scaling of distributed nodes based on actual needs. This approach results in a degree of resource waste. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a method and system for automatic scaling of nodes in a distributed storage system. Based on Hadoop, it performs distributed storage and processing, and uses deep learning algorithms to learn and predict traffic trends to automatically scale the number of nodes elastically, ensuring the quality of storage services and effectively saving storage resources in the distributed storage system.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for automatic node scaling in a distributed storage system includes the following steps:

[0007] Acquire network traffic data and create a dataset;

[0008] Build a deep learning network model for regression prediction;

[0009] The deep learning network model is trained using the dataset to obtain a traffic prediction model for network traffic prediction.

[0010] Obtain real-time network traffic, obtain future network traffic through the traffic prediction model, calculate the storage data requirements corresponding to the future network traffic, calculate the required number of Hadoop nodes based on the storage data requirements, if the current number of Hadoop nodes matches the required number of nodes, then the number of Hadoop nodes remains unchanged and this step is repeated; otherwise, adjust the current number of Hadoop nodes to match the required number of nodes and repeat this step.

[0011] Furthermore, the deep learning network model includes an input layer, a first feature extraction layer, a second feature extraction layer, a first GRU layer, a second GRU layer, a loss function layer, and an output layer. The input layer is used to acquire input data. The first feature extraction layer and the second feature extraction layer extract features from the input data to obtain a first feature value and a second feature value, respectively. The first feature value and the second feature value are input into the first GRU layer after dimension addition and dimension transformation. The output of the first GRU layer is sent to the second GRU layer, and the output of the second GRU layer is sent to the output layer. The loss function layer is used to optimize the parameters of the deep learning network model.

[0012] Furthermore, the principle of the deep learning network model is as follows:

[0013]

[0014] x j2 =f(p(x) i ))

[0015] f(*) = max(0, *)

[0016] x j =cat(x) j1 x j2 )

[0017] x t =reshape(x) j )

[0018] y′ i =GRU1(x t )

[0019] y i =GRU2(x t )

[0020]

[0021] Where m is the number of neurons in the input layer, x i Indicates input data, w i，j b represents the weights in the first feature extraction layer. j x represents the threshold in the first feature extraction layer. j1Let x represent the first eigenvalue. j2 Let x represent the second eigenvalue, p() represent the max pooling operation, and x j This represents the sum of the first and second eigenvalues ​​along the specified dimension. `cat()` indicates summation along the specified dimension. t This represents the value of the first and second eigenvalues ​​after dimensional transformation. `reshape()` represents the dimensional transformation. `GRU1()` and `GRU2()` represent the operations of the first and second GRU layers, respectively. `y′` i This represents the output of the first GRU layer, y i represents the output of the second GRU layer, and MAE represents the loss function.

[0022] Furthermore, the parameters of the deep learning network model are determined through optimization using an improved gray wolf algorithm, as follows:

[0023] ① Initialize the wolf pack Where N represents the size of the wolf pack. This represents the position vector of the i-th wolf, where 1 ≤ i ≤ N. The number of dimensions corresponds one-to-one with the number of parameters to be optimized in the deep learning network model, and the coefficient vector is initialized. coefficient vector and attenuation coefficient Determine the maximum number of iterations and the upper and lower bounds of the parameters to be optimized;

[0024] ② Calculate the fitness value of each wolf in the wolf pack, and designate the three wolves with the best fitness values ​​as α wolf, β wolf and δ wolf in turn, and the rest as ω wolf. If the pre-set convergence condition is met, proceed to step ⑤; otherwise, proceed to step ③.

[0025] ③ Update the position of ω Wolf. The position update formula is as follows:

[0026]

[0027]

[0028]

[0029]

[0030] in, This represents the position of the individual gray wolf in the t-th iteration. Let represent the coefficient vector of α wolf, β wolf, and δ wolf. This represents the position of α wolf, β wolf, and δ wolf in the t-th iteration. This represents the distance between the individual gray wolf and wolf α at the t-th iteration. This represents the distance between the individual gray wolf and the β wolf at the t-th iteration. Let f(*) represent the distance between the gray wolf and the δ wolf at the t-th iteration, where the subscripts i = 1, 2, and 3 correspond to the α wolf, β wolf, and δ wolf, respectively, and f(*) represents the fitness function.

[0031] ④ Update the coefficient vector coefficient vector Perform step ② and update the formula as follows:

[0032]

[0033]

[0034]

[0035] Where iter represents the current iteration number, and Max_iter represents the maximum iteration number. and These represent vectors randomly generated in the range [0, 1].

[0036] ⑤ The position of the α wolf is used as the optimal parameter value for the deep learning network model.

[0037] Furthermore, the first GRU layer and the second GRU layer have the same structure. The parameters to be optimized in the deep learning network model include the number of neurons in the first GRU layer and the size of the convolution kernel.

[0038] Furthermore, the fitness value is calculated as follows: Substitute the parameters corresponding to the gray wolf's position vector into the deep learning network model, train the deep learning network model using the dataset, calculate the loss function in each training iteration, and obtain the fitness value based on the loss function. The formula for calculating the fitness value is:

[0039]

[0040] Where ε is the set hyperparameter, K represents the final number of training iterations of the deep learning network model, and MAE k Let represent the loss function for the k-th iteration.

[0041] Furthermore, the output of the traffic prediction model is the predicted network traffic values ​​{f1, f2, ..., f} within the time period T. N}, where N represents the number of traffic flows. Based on the predicted network traffic values ​​within time period T, the future network traffic is determined as follows:

[0042] Take the peak flow rate f within the time period T. max Take the minimum flow rate f within the time period T. min Take the average flow rate f within the time period T. avg The formula for calculating future network traffic f is as follows:

[0043] f = w1f max +w2f min +w3f avg

[0044] Where w1, w2, and w3 represent weighting coefficients.

[0045] Furthermore, w3 = 0.5, and the formulas for calculating w1 and w2 are as follows:

[0046]

[0047] Where, n h Let {f1, f2, ..., f N} higher than f avg The number of traffic, n l Let {f1, f2, ..., f N} lower than f avg The number of traffic flows.

[0048] Furthermore, the "adaptation of the current number of Hadoop nodes to the required number of nodes" specifically refers to:

[0049] Get the current number of Hadoop nodes D1, get the required number of nodes D2. If D1-λ≤D2≤D1+λ, then the current number of Hadoop nodes is adapted to the required number of nodes, where λ is the preset elastic scaling threshold.

[0050] The specific steps of "adjusting the current number of Hadoop nodes to the required number of nodes" are as follows:

[0051] Obtain the adjustment step size;

[0052] If the current number of Hadoop nodes is less than the required number of nodes, the current number of nodes is increased according to the adjustment step size. It is then determined whether the current number of nodes is adapted to the required number of nodes after the increase. If it is adapted, the adjustment ends; otherwise, this step is repeated.

[0053] If the current number of Hadoop nodes is greater than the required number of nodes, the current number of nodes is reduced by the adjustment step size. It is then determined whether the current number of nodes after the reduction matches the required number of nodes. If it matches, the adjustment ends; otherwise, this step is repeated.

[0054] An auto-scaling system for nodes in a distributed storage system, comprising:

[0055] A traffic prediction module, which is equipped with a trained traffic prediction model, is used to obtain future network traffic based on real-time network traffic.

[0056] The demand node calculation module is used to calculate the storage data demand corresponding to future network traffic, and calculate the number of required nodes for Hadoop based on the storage data demand.

[0057] The adaptation judgment module is used to determine whether the current number of Hadoop nodes is compatible with the required number of nodes;

[0058] The node number adjustment module is used to adjust the current number of nodes in Hadoop;

[0059] The training process of the traffic prediction model is as follows:

[0060] Acquire network traffic data and create a dataset;

[0061] Build a deep learning network model for regression prediction;

[0062] The deep learning network model is trained using the dataset to obtain a traffic prediction model for network traffic prediction.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) By using deep learning algorithms to automatically perceive the changing trend of network traffic data, the number of nodes required by the distributed file system is automatically adjusted according to the changing trend, thereby realizing the dynamic scaling of nodes following network traffic data, ensuring the quality of storage services, and effectively saving storage resources of the distributed storage system.

[0065] (2) The improved Grey Wolf algorithm was used to optimize the parameters of the deep learning network model, which effectively avoided the problems of long training time and large storage space required during model training. At the same time, after optimizing the number of neurons and the size of the convolution kernel of GRU, the prediction accuracy of the model can be effectively improved.

[0066] (3) When calculating future network traffic, the maximum, minimum and average traffic values ​​are taken into account, and the weights are determined based on the number of nodes with traffic values ​​higher than and lower than the average traffic value, making the future network traffic more accurate. Attached Figure Description

[0067] Figure 1 A flowchart for the node auto-scaling method;

[0068] Figure 2 This is a schematic diagram of the structure of a deep learning network model;

[0069] Figure 3 A flowchart for optimizing parameters in the improved Grey Wolf algorithm. Detailed Implementation

[0070] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, providing detailed implementation methods and specific operating procedures. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and the scope of protection of the present invention is not limited to the following embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0071] As used herein, "an embodiment" or "embodiment" refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. Moreover, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0072] This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment), or the execution order of steps without timing constraints can be adjusted.

[0073] Example 1:

[0074] This application provides a method for automatic node scaling in a distributed storage system, such as... Figure 1 As shown, it includes the following steps:

[0075] S1. Obtain network traffic data and create a dataset;

[0076] Acquire network traffic data over a specific period and record relevant information, including year, month, day, weekday, hour, minute, second, millisecond, and the current traffic volume. Determine the training sample length and prediction time step. Truncate the network traffic data according to a certain time length and perform preprocessing operations such as normalization and missing value handling to complete the dataset construction.

[0077] S2. Build a deep learning network model for regression prediction;

[0078] In this application, the constructed deep learning network model includes an input layer, a first feature extraction layer, a second feature extraction layer, a first GRU layer, a second GRU layer, a loss function layer, and an output layer, such as... Figure 2 As shown, the input layer is used to acquire input data. The first feature extraction layer and the second feature layer extract features from the input data to obtain the first feature value and the second feature value, respectively. The first feature value and the second feature value are input into the first GRU layer after dimension addition and dimension transformation. The output of the first GRU layer is sent to the second GRU layer. The output of the second GRU layer is sent to the output layer. The loss function layer is used to optimize the parameters of the deep learning network model.

[0079] In this application, the first feature extraction layer is a max pooling layer, and the second feature extraction layer is a convolutional layer. The first and second GRU layers have the same structure. GRU (Gate Recurrent Unit) is a type of recurrent neural network, which, like LSTM (Long-Short Term Memory), was proposed to solve problems related to long-term memory and gradients in backpropagation. The input-output structure of GRU is the same as that of a regular RNN. Input: Input x at time t. t The hidden layer state H at time t-1 t-1 Output: The output y of the hidden node at time t. t The hidden state h is passed to the next node. t .

[0080] r t =σ(x t W xr +H t-1 W hr +b r )

[0081] z t =σ(x t W xz +H t-1 W hz +b z )

[0082]

[0083]

[0084] y t =W o ·H t

[0085] GRU has only two doors, r tThe update gate, obtained by combining the input gate and forget gate in LSTM into one in GRU, controls the amount of data that can be retained in the current time step, or in other words, determines how much information from the previous time step and the current time step should be passed on to the future; t Reset another door, and control how much past information should be forgotten. H represents the candidate hidden state. t This indicates the final hidden state.

[0086] Specifically, the principles of deep learning network models are as follows:

[0087]

[0088] x j2 =f(p(x) i ))

[0089] f(*) = max(0, *)

[0090] x j =cat(x) j1 x j2 )

[0091] x t =reshape(x) j )

[0092] y′ i =GRU1(x t )

[0093] y i =GRU2(x t )

[0094]

[0095] Where m is the number of neurons in the input layer, x i Indicates input data, w i，j b represents the weights in the first feature extraction layer. j x represents the threshold in the first feature extraction layer. j1 Let x represent the first eigenvalue. j2 Let x represent the second eigenvalue, p() represent the max pooling operation, and x j This represents the sum of the first and second eigenvalues ​​along the specified dimension. `cat()` indicates summation along the specified dimension. t This represents the value of the first and second eigenvalues ​​after dimensional transformation. `reshape()` represents the dimensional transformation. `GRU1()` and `GRU2()` represent the operations of the first and second GRU layers, respectively. `y′` i This represents the output of the first GRU layer, yi The output of the second GRU layer is represented by f(x). MAE represents the loss function. i ) represents the predicted result, y i It indicates the actual result of the record.

[0096] S3. Use the dataset to train the deep learning network model to obtain a traffic prediction model for network traffic prediction.

[0097] Generally, the parameters of deep learning network models are set empirically or randomly, often requiring multiple trials to find the optimal parameters. To find the optimal parameters more quickly, many researchers have proposed using optimization algorithms such as particle swarm optimization and genetic algorithms to optimize parameters and determine the optimal network parameters. Gray Wolf Optimization (GWO) is defined as a swarm intelligence optimization model with advantages such as flexibility, simplicity, and non-derivative mechanisms. Some studies have found that compared with other traditional optimization models, GWO has better numerical properties, can prevent local optima, and is considered a convenient stochastic method for solving highly nonlinear, multivariate, and multimodal optimization problems. GWO is a novel metaheuristic algorithm technique, initially proposed by Miljali et al., and inspired by the hunting and social hierarchy of gray wolves. Gray wolf packs are essentially divided into four classes, named alpha (α), beta (β), delta (σ), and omega (ω). The top level of the hierarchy is the Alpha Wolf. Furthermore, the Alpha Wolf also makes crucial decisions. The second level in the gray wolf hierarchy is the Beta Wolf, which acts as a mentor and provides feedback to empower and assist the Alpha Wolf. On the other hand, Delta wolves obey the commands of Alpha and Beta wolves and dominate Omega wolves. The main phases of the gray wolf hunting process include hunting, surrounding, and attacking the prey. In the GWO algorithm, the hierarchy consists of the best solution, the second best solution, the third best solution, and the remaining candidate solutions.

[0098] However, in practice, the inventors found that directly applying the traditional Grey Wolf algorithm to optimize the parameters of the deep learning network model designed in this application easily leads to getting trapped in local optima, resulting in poor optimization performance. To address this problem, the inventors improved the Grey Wolf algorithm, using the improved algorithm to automatically optimize parameters, making it more reliable and accurate. Figure 3 As shown, the parameter optimization of the deep learning network model using the improved Grey Wolf algorithm is as follows:

[0099] ① Initialize the wolf pack Where N represents the size of the wolf pack. This represents the position vector of the i-th wolf, where 1 ≤ i ≤ N. The number of dimensions corresponds one-to-one with the number of parameters to be optimized in the deep learning network model, and the coefficient vector is initialized. coefficient vector and attenuation coefficient Determine the maximum number of iterations and the upper and lower bounds of the parameters to be optimized; In this embodiment, the parameters to be optimized in the deep learning network model include the number of neurons in the first GRU layer and the second GRU layer and the size of the convolution kernel. The number of gray wolves is set to 50, the maximum number of iterations is set to 100, the lower bound is set to -20, the upper bound is set to 20, and the number of dimensions is set to 3.

[0100] ② Calculate the fitness value of each wolf in the wolf pack, and designate the three wolves with the best fitness values ​​as α wolf, β wolf and δ wolf in turn, and the rest as ω wolf. If the pre-set convergence condition is met, proceed to step ⑤; otherwise, proceed to step ③.

[0101] The fitness value is calculated as follows: Substitute the parameters corresponding to the gray wolf's position vector into the deep learning network model, train the deep learning network model using the dataset, calculate the loss function in each training iteration, and obtain the fitness value based on the loss function. The formula for calculating the fitness value is:

[0102]

[0103] Where ε is the set hyperparameter, K represents the final number of training iterations of the deep learning network model, and MAE k Let represent the loss function for the k-th iteration.

[0104] ③ Update the position of ω Wolf. The position update formula is as follows:

[0105]

[0106]

[0107] In the traditional gray wolf optimization algorithm, the position of ω wolf The update formula is In this application, the new position of the ω wolf in the group is determined based on the fitness scores of the alpha, beta, and delta wolves, as follows:

[0108]

[0109]

[0110]

[0111] in, This represents the position of the individual gray wolf in the t-th iteration. Let represent the coefficient vector of α wolf, β wolf, and δ wolf. This represents the position of α wolf, β wolf, and δ wolf in the t-th iteration. This represents the distance between the individual gray wolf and wolf α at the t-th iteration. This represents the distance between the individual gray wolf and the β wolf at the t-th iteration. Let f(*) represent the distance between the gray wolf and the δ wolf at the t-th iteration, where the subscripts i = 1, 2, and 3 correspond to the α wolf, β wolf, and δ wolf, respectively, and f(*) represents the fitness function.

[0112] ④ Update the coefficient vector coefficient vector Perform step ② and update the formula as follows:

[0113]

[0114]

[0115] in, and Let [0, 1] and [1] represent vectors randomly generated in the range [0, 1]. In the traditional Grey Wolf optimization algorithm, the decay coefficient [1] is... It is a linearly decreasing vector from 2 to 0, although the parameters In the traditional GWO algorithm, the decay rate decreases linearly. However, in many problems, the algorithm's exploration and development behavior needs to change non-linearly to move away from local optima. Accordingly, this application adjusts the decay coefficient. as follows:

[0116]

[0117] Where iter represents the current iteration number, and Max_iter represents the maximum iteration number.

[0118] ⑤ The position of the α wolf is used as the optimal parameter value for the deep learning network model.

[0119] The improvements made to the Grey Wolf algorithm in this application are as follows:

[0120] 1. When updating the position of each gray wolf, instead of averaging the test vectors, each position is updated by weighting the test vectors according to the score weights of the three leader gray wolves. This update mechanism helps to improve the algorithm's exploration and development capabilities, improves the ability to find the optimal parameters of GUR, and thus improves the accuracy of the deep learning network model proposed in this application.

[0121] 2. Attenuation coefficient Adjustments were made. Although the parameter decreased linearly in the original GWO algorithm, in many problems, the algorithm's exploration and development behavior needs to change non-linearly to move away from local optima. To avoid the algorithm getting trapped in local optima, this application redesigned... The update rules.

[0122] With the above two modifications, the improved GWO algorithm is more suitable for optimizing the deep learning network model proposed in this application. Extensive experiments have demonstrated that the improved GWO algorithm outperforms the original GWO algorithm in optimizing the proposed deep learning network model, achieving superior optimization results. The improved algorithm improves the allowable exploration time during the transition process by 30% (the exploration time in the original GWO process accounts for approximately 35% of the total time, while the improved algorithm accounts for 65%), ultimately reducing the mean square error of the prediction results by approximately 1%. Furthermore, it only requires 90% of the iterations of the original GWO algorithm to find the optimal solution.

[0123] In the prior art, although some people have studied traffic prediction algorithms based on deep learning, they are often limited by the prediction ability of the deep learning model itself, and have shortcomings such as low accuracy and difficulty in training the optimal parameters. In the deep learning network model designed in this application, an improved Grey Wolf algorithm is designed. This algorithm can help the deep learning model find the optimal network model parameters for network traffic data, so it can bring the advantages of higher prediction accuracy and more concise model parameters.

[0124] Network traffic data changes contain a wealth of information, which can be interpreted as whether the network is under attack or as the amount of tasks that need to be processed on the network. As the network continues to develop, network traffic changes become more and more complex and this characteristic becomes more and more difficult to understand. Therefore, this application uses deep learning to deeply mine the potential information related to traffic data changes and tasks, thereby performing traffic prediction.

[0125] Understandably, in the training of deep learning network models, the difference between the predicted traffic sequence and the actual traffic sequence can be used to judge whether the deep learning network model is effective. The dataset can also be divided into training set, validation set, test set, etc., which will not be elaborated here.

[0126] S4. Obtain real-time network traffic, obtain future network traffic through the traffic prediction model, calculate the storage data requirements corresponding to the future network traffic, and calculate the required number of Hadoop nodes based on the storage data requirements. If the current number of Hadoop nodes matches the required number of nodes, the number of Hadoop nodes remains unchanged, and this step is repeated. Otherwise, the current number of Hadoop nodes is adjusted to match the required number of nodes, and this step is repeated.

[0127] The output of the traffic prediction model is the predicted network traffic values ​​{f1, f2, ..., f} within the time period T. N}, where N represents the number of traffic flows. For example, if T can be 1 second and the time granularity is set to 1 ms, then the number of traffic flows N is 1000. Each time granularity has a traffic prediction value. Based on the network traffic prediction values ​​within the time period T, the future network traffic is determined as follows:

[0128] Take the peak flow rate f within the time period T. max Take the minimum flow rate f within the time period T. min Take the average flow rate f within the time period T. avg The formula for calculating future network traffic f is as follows:

[0129] f = w1f max +w2f min +w3f avg

[0130] Where w1, w2, and w3 represent weighting coefficients.

[0131] w3 = 0.5, the formulas for calculating w1 and w2 are:

[0132]

[0133] Where, n h Let {f1, f2, ..., f N} higher than f avg The number of traffic, n l Let {f1, f2, ..., f N} lower than f avg The number of traffic flows.

[0134] Where, n h Let {f1, f2, ..., f N} higher than f avg The number of traffic, n l Let {f1, f2, ..., f N} lower than f avg The calculation of future network traffic considers the maximum, minimum, and average traffic values, and assigns weights based on the number of nodes with traffic values ​​above and below the average, thus making the calculation of future network traffic more accurate.

[0135] The determination of "the current number of Hadoop nodes matches the required number of nodes" is as follows: obtain the current number of Hadoop nodes D1, obtain the required number of nodes D2, and if D1-λ≤D2≤D1+λ, then the current number of Hadoop nodes matches the required number of nodes. Here, λ is a preset elastic scaling threshold. In this embodiment, λ is set to 3 / 4 of the current number of Hadoop nodes.

[0136] The process of "adjusting the current number of Hadoop nodes to match the required number of nodes" is as follows: Obtain the adjustment step size; if the current number of Hadoop nodes is less than the required number of nodes, increase the current number of nodes according to the adjustment step size, and determine whether the increased current number of nodes matches the required number of nodes. If it matches, the adjustment ends; otherwise, repeat this step. If the current number of Hadoop nodes is greater than the required number of nodes, decrease the current number of nodes according to the adjustment step size, and determine whether the decreased current number of nodes matches the required number of nodes. If it matches, the adjustment ends; otherwise, repeat this step.

[0137] The step size can be set to an appropriate size to gradually increase or decrease the number of nodes until the requirements are met.

[0138] This invention is the first to analyze network traffic to predict the demand for data on distributed storage nodes and automatically expand the nodes. At the same time, in order to save resources and not affect storage needs, some scaling results will be adjusted again, truly realizing dynamic scaling of distributed nodes and minimizing the waste of node resources.

[0139] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0140] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.

[0141] This application also provides an automatic node scaling system for a distributed storage system, including:

[0142] The traffic prediction module is equipped with a pre-trained traffic prediction model, which is used to obtain future network traffic based on real-time network traffic.

[0143] The demand node calculation module is used to calculate the storage data demand corresponding to future network traffic, and calculate the number of required nodes for Hadoop based on the storage data demand.

[0144] The adaptation judgment module is used to determine whether the current number of Hadoop nodes is compatible with the required number of nodes;

[0145] The node number adjustment module is used to adjust the current number of nodes in Hadoop;

[0146] The training process for the traffic prediction model is as follows:

[0147] Acquire network traffic data and create a dataset;

[0148] Build a deep learning network model for regression prediction;

[0149] The deep learning network model is trained using the dataset to obtain a traffic prediction model for network traffic prediction.

[0150] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for automatic node scaling in a distributed storage system, characterized in that, Includes the following steps: Acquire network traffic data and create a dataset; Build a deep learning network model for regression prediction; The deep learning network model is trained using the dataset to obtain a traffic prediction model for network traffic prediction. Obtain real-time network traffic, obtain future network traffic through the traffic prediction model, calculate the storage data requirements corresponding to the future network traffic, calculate the required number of Hadoop nodes based on the storage data requirements, if the current number of Hadoop nodes matches the required number of nodes, then the number of Hadoop nodes remains unchanged and this step is repeated; otherwise, adjust the current number of Hadoop nodes to match the required number of nodes and repeat this step. The deep learning network model includes an input layer, a first feature extraction layer, a second feature extraction layer, a first GRU layer, a second GRU layer, a loss function layer, and an output layer. The input layer is used to acquire input data. The first feature extraction layer and the second feature extraction layer extract features from the input data to obtain a first feature value and a second feature value, respectively. The first feature value and the second feature value are then input into the first GRU layer after dimension addition and dimension transformation. The output of the first GRU layer is sent to the second GRU layer, and the output of the second GRU layer is sent to the output layer. The loss function layer is used to optimize the parameters of the deep learning network model. The principle of the deep learning network model is as follows: in, The number of neurons in the input layer. Indicates input data, This represents the weights in the first feature extraction layer. This represents the threshold in the first feature extraction layer. Represents the first eigenvalue. Represents the second eigenvalue. This indicates a max pooling operation. This represents the value obtained by adding the first and second eigenvalues ​​together according to their dimensions. This indicates addition along the dimension. This represents the value of the first and second eigenvalues ​​after dimensional transformation. Indicates dimensional transformation. and These represent the operations of the first GRU layer and the second GRU layer, respectively. This represents the output of the first GRU layer. This represents the output of the second GRU layer. Represents the loss function; The parameters of the deep learning network model are determined through optimization using an improved Grey Wolf algorithm, as follows: ① Initialize the wolf pack { },in, Indicates the size of the wolf pack. Indicates the first The position vectors of the wolves. , The number of dimensions corresponds one-to-one with the number of parameters to be optimized in the deep learning network model, and the coefficient vector is initialized. coefficient vector and attenuation coefficient Determine the maximum number of iterations and the upper and lower bounds of the parameters to be optimized; ② Calculate the fitness value of each wolf in the wolf pack, and designate the three wolves with the best fitness values ​​as follows: Wolf, wolves and Wolves, the rest are designated as If the wolf meets the preset convergence condition, proceed to step ⑤; otherwise, proceed to step ③. ③Update The wolf's position, updated using the following formula: in, Indicates the number of individual gray wolves The position of the next iteration. , , , , , express Wolf, wolves and The coefficient vector of the wolf. , , express Wolf, wolves and Wolf The position of the next iteration. Indicates the first In the next iteration, the gray wolf individual and The distance between wolves Indicates the first In the next iteration, the gray wolf individual and The distance between wolves Indicates the first In the next iteration, the gray wolf individual and Distance between wolves, subscript 1, 2, and 3 correspond to respectively Wolf, wolves and Wolf, Represents the fitness function; ④ Update the coefficient vector coefficient vector Perform step ② and update the formula as follows: in, Indicates the current iteration number. Indicates the maximum number of iterations. and These represent vectors randomly generated in the range [0,1]. ⑤ The wolf's position serves as the optimal parameter value for the deep learning network model.

2. The method for automatic node scaling in a distributed storage system according to claim 1, characterized in that, The first and second GRU layers have the same structure. The parameters to be optimized in the deep learning network model include the number of neurons in the first and second GRU layers and the size of the convolution kernel.

3. The method for automatic node scaling in a distributed storage system according to claim 2, characterized in that, The fitness value is calculated as follows: Substitute the parameters corresponding to the gray wolf's position vector into the deep learning network model, train the deep learning network model using the dataset, calculate the loss function in each training iteration, and obtain the fitness value based on the loss function. The formula for calculating the fitness value is: in, For setting hyperparameters, This indicates the final number of training iterations for the deep learning network model. Indicates the first The loss function for the next iteration.

4. The method for automatic node scaling in a distributed storage system according to claim 1, characterized in that, The output of the traffic prediction model is the predicted network traffic value within the time period T. }, The future network traffic is determined based on the predicted network traffic values ​​within time period T, representing the traffic volume. Take the peak traffic volume within time period T Take the minimum flow rate within time period T. Take the average flow rate within the time period T. Then future network traffic The calculation formula is as follows: in, , and This represents the weighting coefficient.

5. The method for automatic node scaling in a distributed storage system according to claim 4, characterized in that, 0.5, , The calculation formula is: in, for{ } higher than The number of traffic, for{ } lower than The number of traffic flows.

6. The method for automatic node scaling in a distributed storage system according to claim 1, characterized in that, The "hadoop current node number adapts to the required node number" specifically refers to: Get the current number of Hadoop nodes Get the number of required nodes ,like ≤ ≤ Then the current number of Hadoop nodes is adapted to the required number of nodes, where, The preset elastic scaling threshold; The specific steps of "adjusting the current number of Hadoop nodes to the required number of nodes" are as follows: Obtain the adjustment step size; If the current number of Hadoop nodes is less than the required number of nodes, the current number of nodes is increased according to the adjustment step size. It is then determined whether the current number of nodes is adapted to the required number of nodes after the increase. If it is adapted, the adjustment ends; otherwise, this step is repeated. If the current number of Hadoop nodes is greater than the required number of nodes, the current number of nodes is reduced by the adjustment step size. It is then determined whether the current number of nodes after the reduction matches the required number of nodes. If it matches, the adjustment ends; otherwise, this step is repeated.

7. A system for implementing the node autoscaling method of the distributed storage system as described in claim 1, characterized in that, include: A traffic prediction module, which is equipped with a trained traffic prediction model, is used to obtain future network traffic based on real-time network traffic. The demand node calculation module is used to calculate the storage data demand corresponding to future network traffic, and calculate the number of required nodes for Hadoop based on the storage data demand. The adaptation judgment module is used to determine whether the current number of Hadoop nodes is compatible with the required number of nodes; The node number adjustment module is used to adjust the current number of nodes in Hadoop; The training process of the traffic prediction model is as follows: Acquire network traffic data and create a dataset; Build a deep learning network model for regression prediction; The deep learning network model is trained using the dataset to obtain a traffic prediction model for network traffic prediction.

Images