A deep learning dynamic weight adjustment method applied to hyper-converged storage pool

Abstract

The application discloses a kind of deep learning dynamic weight adjustment methods applied to super-converged storage pool, belong to deep learning technical field, including the following steps: S1, create super-converged cluster, S2, create corresponding data pool according to proportion, S3, the use of each data pool in a period of time is collected, S4, the feature collected is trained and predicted using time series three-layer convolutional neural network.Utilize time series convolutional neural network in deep learning as the deep learning dynamic weight of super-converged storage pool provides adjustment scheme, weight adjustment is added to the storage cluster of super-convergence, while guaranteeing performance in controlling cost and diversity, through the prediction means of deep learning, data pool weight is dynamically adjusted in real time, data balance is strengthened, avoid weight excessive adjustment, standardized means is used to handle and obtain adjustment weight parameter, real-time prediction of user demand for different storage types is realized, meet the intelligent storage scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a deep learning dynamic weight adjustment method applied to hyper-converged storage pool. BACKGROUND

[0002] With the rapid development of modern information technology, cloud computing services are becoming increasingly important, and distributed storage technology has always been one of the research hotspots in the field of cloud computing. Distributed storage technology inevitably affects the upper-layer cloud computing service interface. The innovation of storage technology is of great significance to the development of cloud computing. Storage is an indispensable part of infrastructure as a service. As an important foundation of cloud services, storage technology faces many customized storage needs, and the more diverse processing and cost control become new challenges.

[0003] Hyper-converged storage is an aspect of hyper-converged infrastructure (HCI), in which storage is bundled with compute and network in a virtualized system, using a flexible storage pool instead of dedicated hardware through a software-defined approach. Each node contains a software layer that virtualizes and shares resources among all nodes in the cluster, creating a large storage pool, which is the hyper-converged storage pool. Hyper-converged storage enables administrators to more easily manage resources and reduce the total cost of ownership of storage, and in many cases, it can ensure better storage pricing compared to public cloud service providers. In addition, hyper-converged storage has some significant advantages, such as providing greater flexibility, reducing IT operating expenses, and saving storage costs. However, it also has some disadvantages, such as limited scalability, the need for additional organizational planning to increase implementation complexity, etc.

[0004] Deep learning (DL) is a new research direction in the field of machine learning (ML), which is introduced into machine learning to make it closer to the original goal-artificial intelligence (AI). Deep learning is to learn the internal rules and representation levels of sample data, and the information obtained in the learning process is very helpful for the interpretation of data such as text, images and sound. The ultimate goal is to enable machines to have analysis and learning ability like people, and to recognize data such as text, images and sound. Deep learning is a complex machine learning algorithm, which has achieved better results in speech and image recognition than previous related technologies. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as vision and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies. Deep learning can be understood as the combination of the two nouns "deep" and "learning". "Deep" is reflected in the number of layers of neural networks. Generally speaking, the more layers of neural networks, that is, the deeper, the better the learning effect. "Learning" reflects that neural networks can automatically correct parameters such as weight bias by continuously irrigating data to fit better learning effect. The current mainstream deep learning frameworks are TensorFlow, Keras, Caffe, PyTorch, etc. What is the relationship between deep learning and neural network? Deep learning is a deep neural network with deep layers. In the field of deep learning, there are all kinds of neural networks, among which the most famous is convolutional neural network (CNN). What is the relationship between neural network and perception machine? Perception machine is the origin of neural network, and neural network is a perception machine using nonlinear activation function (such as sigmoid function).

[0005] In deep learning, the initial value of the weight has an important influence on the learning performance and speed of the neural network. Generally, the initial value of the weight is randomly set in a small range, which can speed up the convergence and prevent overfitting. However, if the initial value of the weight is too small, it may cause the network to fail to learn correctly; if the initial value of the weight is too large, it may cause the network to learn slowly. To solve this problem, one method is to use weight decay to suppress the occurrence of overfitting. Weight decay is a method of learning with the purpose of reducing the value of the weight parameter, which suppresses the occurrence of overfitting by reducing the value of the weight parameter. During training, the weight value can be updated by the backpropagation algorithm, and a small learning rate can be used to gradually adjust the weight. In addition, regularization methods such as L1 and L2 regularization can also help control the size of the weight and prevent overfitting. Regularization methods limit the complexity of the model and improve the generalization ability of the model by adding some constraints to the weight parameters (such as limiting the absolute value of the weight). When selecting weights, it needs to be decided whether to use regularization methods and the size of the regularization coefficient according to the actual situation. In addition, learning rate adjustment is also an important skill in deep learning. A larger learning rate can help the model converge faster, but it may also cause the model to fall into a local optimal solution. Therefore, the learning rate needs to be dynamically adjusted according to the performance during training. In the early stage of training, a larger learning rate can be used to speed up convergence, while in the later stage of training, the learning rate can be gradually reduced to avoid overfitting. In summary, the selection of weights needs to be determined according to the specific task and data, and needs to be constantly tried and adjusted to obtain the best performance and generalization ability.

[0006] In recent years, artificial intelligence technology has developed rapidly, and deep learning technology has shown extraordinary ability in natural language processing, autonomous driving, medical image processing and other fields. Time series prediction is inspired by the development of more and more deep learning models based on convolutional neural networks, recurrent neural networks and attention models, and has been widely applied in meteorology and finance.

[0007] Distributed storage technology has good scalability and scalability compared to centralized storage technology. Distributed storage with no central architecture can provide different storage services suitable for different storage needs: block storage, file storage and object storage. With the popularity of cloud services in enterprises, the demand for service diversity and cost sensitivity has increased. It is particularly important to ensure diversity and cost while not sacrificing performance. The hyper-converged storage cluster can meet the different storage coexistence of small clusters, and the reasonable adjustment of the data pool weight can balance the performance impact of data throughput in the cluster. Therefore, the dynamic adjustment method of the data pool weight of the hyper-converged storage cluster needs to be solved. SUMMARY

[0008] The application aims to provide a storage service that can meet diversified storage requirements, reduce costs, ensure performance and be intelligent, and proposes a deep learning dynamic weight adjustment method applied to a hyper-converged storage pool.

[0009] To achieve the above effects, the application provides the following technical scheme: a deep learning dynamic weight adjustment method applied to a hyper-converged storage pool, comprising the following steps:

[0010] S1, creating a hyper-converged cluster.

[0011] S2, creating corresponding data pools in proportion.

[0012] S3, collecting the usage of each data pool in a period of time.

[0013] S4, training and predicting the collected features using a time series three-layer convolutional neural network.

[0014] S5, predicting feature values.

[0015] S6, concatenating the predicted feature vectors into a feature matrix and classifying different features.

[0016] S7, determining the adjusted data pool weight according to the predicted feature values.

[0017] S8, dynamically adjusting the weight of each pool in real time according to the obtained weight value,

[0018] Further, the following steps are included: according to the operation steps in S1 and S2, the hyper-converged cluster is created, and corresponding data pools are created in a manner of equally dividing the storage space of block storage, file storage and object storage.

[0019] Further, the following steps are included: according to the operation steps in S3, the usage of each data pool in a period of time is collected during the use of the cluster, including: the usage of the data pool, the growth rate of the usage of the data pool, the total number of objects in the data pool, the number of unhealthy objects in the data pool, the proportion of the number of unhealthy objects in the data pool in the total number of objects, the number of degraded objects in the data pool, the proportion of the number of degraded objects in the data pool in the total number of objects, the number of objects misplaced in the data pool, the proportion of the number of misplaced objects in the data pool in the total number of objects, the number of lost objects in the data pool, the proportion of the number of lost objects in the data pool in the total number of objects, the number of objects recovered per second, the rate of read operation and the rate of write operation.

[0020] Further, the following steps are included: according to the operation steps in S4, the above features are trained and predicted using the timing three-layer convolutional neural network, and the activation function formula is as follows:

[0021]

[0022] Where T represents time, l represents the lth layer of convolution, T represents the lth layer of convolution kernel at T, i represents the previous i time sampling point, W(l, x) is the convolution weight vector of the lth layer, That is, the activation function of the network.

[0023] Further, the following steps are included: according to the operation steps in S6, the predicted feature vectors are spliced into a feature matrix, and different features are classified.

[0024] Further, the following steps are included: according to the operation steps in S6, the data pool usage, the growth rate of the data pool usage, the total number of objects in the data pool, the number of unhealthy objects in the data pool, the proportion of the number of unhealthy objects in the data pool to the total number of objects, the number of degraded objects in the data pool, the proportion of the number of degraded objects in the data pool to the total number of objects, the number of objects misplaced in the data pool, the proportion of the number of misplaced objects in the data pool to the total number of objects, the number of lost objects in the data pool, and the proportion of the number of lost objects in the data pool to the total number of objects are A group.

[0025] Further, the following steps are included: according to the operation steps in S6, the number of objects recovered per second, the rate of read operation, and the proportion of data pool healthy objects are B group.

[0026] Further, the following steps are included: according to the operation steps in S7, the adjusted data pool weight is determined according to the predicted feature value

[0027]

[0028] Further, the following steps are included: according to the operation steps in S7, m represents the total number of features taken, n represents the total number of pools, p n represents the nth pool, represents the p n th pool, n predicted feature value of m features in the pth pool, α m represents the weight coefficient of m features, the coefficient weight of A group is -1, and the coefficient weight of B group is 1, That is, the weight of the acquired p1 data needs to be adjusted.

[0029] Further, the following steps are included: according to the operation steps in S8, the weight vector is Z-Score standardized to obtain the final adjustment weight, and the formula is as follows:

[0030]

[0031] wherein std(w) is the standard deviation of all weight values obtained in the previous step, mean(w) is the mean of all weight values obtained in the previous step, is the final obtained pool weight.

[0032] The application provides a deep learning dynamic weight adjustment method applied to a hyper-converged storage pool, which has the following beneficial effects:

[0033] The time sequence convolutional neural network in deep learning provides an adjustment scheme for the deep learning dynamic weight of the hyper-converged storage pool, and the weight adjustment is added to the hyper-converged storage cluster, thereby ensuring the performance while controlling the cost and diversity, dynamically adjusting the data pool weight in real time through the prediction means of deep learning, strengthening the data balance, avoiding excessive weight adjustment, using the standardization means to process and obtain the adjustment weight parameter, realizing real-time prediction of the demand of users for different storage types, and meeting the intelligent storage scheme. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 FIG. 1 is a flowchart of a deep learning dynamic weight adjustment method applied to a hyper-converged storage pool according to the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to specific embodiments.

[0036] Embodiment 1: A deep learning dynamic weight adjustment method applied to a hyper-converged storage pool, comprising the following steps:

[0037] Step one, creating a hyper-converged cluster, creating corresponding data pools in the manner of dividing storage space according to block storage, file storage and object storage.

[0038] Step two, collecting the usage of each data pool in a period of time during the use of the cluster, including: the amount of data pool, the growth rate of data pool amount, the total number of data pool objects, the number of unhealthy objects in the data pool, the proportion of the number of unhealthy objects in the data pool to the total number of objects, the number of degraded objects in the data pool, the proportion of the number of degraded objects in the data pool to the total number of objects, the number of incorrectly placed objects in the data pool, the proportion of the number of incorrectly placed objects in the data pool to the total number of objects, the number of lost objects in the data pool, the proportion of the number of lost objects in the data pool to the total number of objects, the number of objects recovered per second, the rate of read operation and the rate of write operation.

[0039] Step three, using a time sequence three-layer convolutional neural network to train and predict the above features, and the activation function formula is as follows:

[0040]

[0041] Where T represents time, and l represents the lth layer of convolution. W(l,i) represents the lth layer of convolution kernel at time T. i represents the i th time sample point, and W(l,x) is the convolution weight vector of the lth layer.

[0042] That is the activation function of the network.

[0043] Step four, according to the trained convolution weight to predict the feature value in step 2.

[0044] Step five, the predicted feature vector is spliced into a feature matrix, and different features are classified, wherein the amount of data pool, the growth rate of data pool amount, the total number of data pool objects, the number of unhealthy objects in data pool, the proportion of the number of unhealthy objects in data pool to the total number of objects, the number of degraded objects in data pool, the proportion of the number of degraded objects in data pool to the total number of objects, the number of objects misplaced in data pool, the proportion of the number of misplaced objects in data pool to the total number of objects, the number of lost objects in data pool, and the proportion of the number of lost objects in data pool to the total number of objects are A group. The number of objects recovered per second, the rate of read operation, and the proportion of data pool healthy objects are B group.

[0045] Step six, according to the predicted feature value to determine the adjusted data pool weight

[0046]

[0047] Where m represents the total number of features taken, n represents the total number of pools, and p n The nth pool, The pth pool, n The predicted feature value of m features in the pool, and α m The weight coefficient of m features. The coefficient weight of A group is-1, and the coefficient weight of B group is 1, That is, the weight of the acquired p1 data needs to be adjusted.

[0048] Step seven, Z-Score standardization is performed on the weight vector to obtain the final adjustment weight, and the formula is as follows:

[0049]

[0050] Where std(w) is the standard deviation of all weight values obtained in the previous step, and mean(w) is the mean value of all weight values obtained in the previous step, Z-Score standardization, also known as standard deviation standardization, is a commonly used data preprocessing method for the final obtained pool weight. It converts the original data into Z-Score scores by means of (x-μ) / σ, that is, it converts data of different magnitudes into Z-Score scores of uniform magnitude for comparison. This method is suitable for cases where the maximum and minimum values of attribute A are unknown or there are outlier data beyond the value range. The processed data conforms to the standard normal distribution, that is, the mean is 0 and the standard deviation is 1.

[0051] Step eight, according to the obtained weight value, dynamically adjust the weight of each pool.

[0052] Embodiment 2: a deep learning dynamic weight adjustment method applied to hyper-converged storage pools, comprising the following steps:

[0053] Step one, build a hyper-converged cluster, and create data pools in a ratio of one-third block storage, one-third file storage, and one-third object storage.

[0054] Step two, record the read and write rates of the existing running cluster data every five minutes for two weeks, and refer to the recorded data to regularly feed data into the cluster.

[0055] Step three, simultaneously collect the usage of each data pool in the first week, including: data pool usage, data pool usage growth rate, total number of data pool objects, number of unhealthy objects in the data pool, proportion of unhealthy objects in the data pool, number of degraded objects in the data pool, proportion of degraded objects in the data pool, number of incorrectly placed objects in the data pool, proportion of incorrectly placed objects in the data pool, number of lost objects in the data pool, proportion of lost objects in the data pool, number of objects recovered per second, read operation rate, and write operation rate.

[0056] Step four, use the above features collected in the first week to train and predict the time series three-layer convolutional neural network.

[0057] Step five, use the trained network model to predict the feature prediction.

[0058] Step six, real-time acquisition of weight parameters for data pool weight adjustment.

[0059] Step seven, the alarm ratio of the cluster health state is reduced by about 15% compared with the previous state.

[0060] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A deep learning dynamic weight adjustment method applied to hyperconverged storage pools, characterized in that, Includes the following steps: S1. Create a hyperconverged cluster and create corresponding data pools by equally allocating storage space for block storage, file storage, and object storage. S2. During cluster usage, collect the usage data of each data pool over a period of time, including: data pool usage, data pool usage growth rate, total number of data pool objects, number of unhealthy objects in the data pool, percentage of unhealthy objects in the data pool to the total number of objects, number of degraded objects in the data pool, percentage of degraded objects in the data pool to the total number of objects, number of misplaced objects in the data pool, percentage of misplaced objects in the data pool to the total number of objects, number of lost objects in the data pool, percentage of lost objects in the data pool to the total number of objects, number of objects recovered per second, read operation rate, and write operation rate. S3. Use a three-layer temporal convolutional neural network to predict and train the collected features; S4. Predict the feature values ​​in step S2 based on the trained convolution weights; S5. Concatenate the predicted feature values ​​into a feature matrix and classify different features. Group A includes: data pool usage, data pool usage growth rate, total number of data pool objects, number of unhealthy objects in the data pool, proportion of unhealthy objects in the data pool to the total number of objects, number of downgraded objects in the data pool, proportion of downgraded objects in the data pool to the total number of objects, number of misplaced objects in the data pool, proportion of misplaced objects in the data pool to the total number of objects, number of lost objects in the data pool, and proportion of lost objects in the data pool to the total number of objects. Group B includes: number of objects recovered per second, read operation rate, and percentage of healthy objects in the data pool. S6. Determine the adjusted data pool weights based on the predicted feature values. Where m represents the total number of features taken, n represents the total number of pools, and p n Let t represent the nth pool. mpn Indicates the p-th n The predicted feature value of feature m in the pool, α m This represents the weight coefficient of feature m. The weight coefficient for group A is -1, and the weight coefficient for group B is 1. This indicates the weights that need to be adjusted based on the acquired p1 data; S7. Perform Z-score standardization on the weight vector to obtain the final adjustment weights, as shown in the following formula: Where std(w) is the standard deviation of all weight values ​​obtained in the previous step, and mean(w) is the mean of all weight values ​​obtained in the previous step. This is the final pool weight; S8. Based on the obtained weight values, dynamically adjust the weights of each pool in real time.

2. The deep learning dynamic weight adjustment method applied to a hyperconverged storage pool according to claim 1, characterized in that, Includes the following steps: According to the operation steps in S3, the collected features are used to perform prediction training using a three-layer temporal convolutional neural network, and the activation function formula is as follows: Where T represents time, l represents the l-th convolutional layer, represents the l-th convolutional kernel at time T, i represents the i-th sampling points at previous time points, and W(l,x) is the convolutional weight vector of the l-th layer. This is the activation function of the network.

Images