A data recovery method, device and storage medium

Abstract

The application discloses a data recovery method, device and storage medium. The method comprises the following steps: acquiring first index data of a storage cluster in a current time period, wherein the first index data comprises data of at least one performance index of the storage cluster; based on the first index data, predicting to obtain predicted bandwidth data of the storage cluster in a next time period, wherein the predicted bandwidth data is used for representing bandwidth resource usage of the storage cluster; based on the predicted bandwidth data, determining a bandwidth allocation amount of data recovery in the next time period, so as to allocate bandwidth resources of the storage cluster in the next time period according to the bandwidth allocation amount for data recovery. Through the above method, the timeliness of data recovery can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage management, and particularly relates to a data recovery method, device and storage medium. BACKGROUND

[0002] In the technical field of data storage management, storage services are often accompanied by huge data storage, and various software and hardware problems such as disk damage, memory failure, file system error, operating system vulnerability, application program crash and network interruption faced by the system itself can all become potential risks of data damage. In order to effectively cope with the frequent data damage challenges in the distributed storage cluster and ensure the reliability and security of the storage data, a series of data recovery methods need to be taken. However, the existing data recovery methods are mostly limited to the instant judgment and response of the cluster state, lack of forward-looking scheduling, which may lead to the failure of timely data recovery, resulting in data loss and reducing the timeliness of data recovery. SUMMARY

[0003] The technical problem solved by the present application is to provide a data recovery method, device and storage medium, which can improve the timeliness of data recovery.

[0004] To solve the above technical problem, one technical solution adopted by the present application is to provide a data recovery method, which comprises: acquiring first index data of the storage cluster in a current time period, wherein the first index data comprises data of at least one performance index of the storage cluster; based on the first index data, predicting to obtain predicted bandwidth data of the storage cluster in a next time period, wherein the predicted bandwidth data is used to represent the bandwidth resource usage of the storage cluster; based on the predicted bandwidth data, determining a bandwidth allocation amount of data recovery in the next time period, so as to allocate the bandwidth resources of the storage cluster in the next time period according to the bandwidth allocation amount for data recovery.

[0005] To solve the above technical problem, another technical solution adopted by the present application is to provide an electronic device, which comprises a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to realize the above-mentioned data recovery method.

[0006] To solve the above technical problem, still another technical solution adopted by the present application is to provide a computer readable storage medium, which is used to store program instructions, and the program instructions can be executed by a processor to realize the above-mentioned data recovery method.

[0007] The scheme is configured to obtain first index data of the storage cluster in a current time period. Based on the first index data, predicted bandwidth data of the storage cluster in a next time period is obtained. Based on the predicted bandwidth data, a bandwidth allocation amount for data recovery in the next time period is determined, so that bandwidth resources of the storage cluster are allocated for data recovery in the next time period according to the bandwidth allocation amount. The present application predicts the predicted bandwidth data of the storage cluster in the next time period by using the data of at least one performance index of the storage cluster in the current time period, and further determines the bandwidth allocation amount for data recovery in the next time period, so that the bandwidth allocation amount can be used for data recovery in time, thereby improving the timeliness of data recovery. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a flowchart of an embodiment of the data recovery method provided by the present application;

[0009] Figure 2 is a training flowchart of a bandwidth prediction model of the data recovery method provided by the present application;

[0010] Figure 3 is a partial flowchart of an embodiment of the data recovery method provided by the present application;

[0011] Figure 4 is a flowchart of an embodiment of the data recovery device provided by the present application;

[0012] Figure 5 is a framework diagram of an embodiment of the electronic device provided by the present application;

[0013] Figure 6 is a framework diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0014] To make the purpose, technical solutions and effects of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and embodiments.

[0015] It should be noted that the term "several" herein represents at least one, and the terms "first", "second", etc. are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. The term "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects. In addition, the term "at least one" herein represents any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C, which can represent any one or more elements selected from the set consisting of A, B, and C.

[0016] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the data recovery method provided by the present application. It should be noted that the embodiment is not limited to the flow order shown in Figure 1 . As shown in Figure 1 , the embodiment includes:

[0017] Step S11: Obtain first index data of the storage cluster in the current time period.

[0018] Among them, the first index data includes data of at least one performance index of the storage cluster.

[0019] In an embodiment, for each performance index in the at least one performance index, the data corresponding to the performance index at different time points in the current time period is collected to obtain the first index data, wherein the first index data includes a multi-dimensional vector corresponding to each time point, and the multi-dimensional vector includes data of the at least one performance index at the corresponding time point.

[0020] In a specific embodiment, the at least one performance index includes at least one of total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster business, and network delay.

[0021] For example, the total bandwidth, the total write bandwidth, the total read bandwidth, the total data recovery bandwidth, the CPU usage, the memory usage, the disk input / output, the total IOPS of cluster service, and the network delay are extracted from the total bandwidth, the total write bandwidth, the total read bandwidth, the total data recovery bandwidth, the CPU usage, the memory usage, the disk input / output, the total IOPS of cluster service, and the network delay as the performance index data to be collected by the storage cluster in the current time period. For the selected total bandwidth, the total write bandwidth, the total read bandwidth, the total data recovery bandwidth, and the CPU usage, a collection period is set in minutes, and the performance index data is sampled according to the collection period. The data corresponding to different time points in the current time period is collected, the index data corresponding to each time point is vectorized to form a multi-dimensional vector corresponding to each time point. Specifically, the first index data includes the multi-dimensional vector corresponding to each time point.

[0022] In another embodiment, considering that the first index data obtained may contain abnormal noise data or repetitive redundant data without information content, before predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the first index data can also be data cleaned to obtain filtered first index data.

[0023] For example, the mathematical expectation of each performance index in the first index data is calculated and the standard deviation , and For the first index data outside the interval, denoising processing is performed, where the mathematical expectation and the standard deviation Z represents the first index data. Of course, for This range can be changed according to actual needs, which is not limited here.

[0024] In another embodiment, considering that the first index data composed of multiple performance indexes of different dimensions has high data complexity and is difficult to be statistically analyzed, before predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the data corresponding to the performance indexes of different dimensions in the first index data can also be processed by dimensionless.

[0025] In a specific embodiment, the data normalization processing method is used to uniformly convert the data of performance indexes of different dimensions into data corresponding to dimensionless performance indexes. For example, the Z-Score standardization method is used to obtain the processed first index data. Since the data corresponding to each performance index is close to a normal distribution, the dimensionless processed first index data can be obtained by where B represents the dimensionless processed first index data, and A represents the obtained first index data.

[0026] In yet another embodiment, before predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the first index data can also be subjected to data cleaning and dimensionless processing of different dimension data in the first index data.

[0027] For example, the first index data can be subjected to data cleaning first, and then the first index data subjected to data cleaning is subjected to dimensionless processing. The specific processing manner can be referred to the related description in step S11. It should be noted that, at this time, the first index data subjected to dimensionless processing is utilized to predict the predicted bandwidth data of the storage cluster in the next time period. The first index data subjected to dimensionless processing is obtained, and A refers to the first index data subjected to data cleaning.

[0028] Step S12: predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data.

[0029] The predicted bandwidth data is used to represent the bandwidth resource usage of the storage cluster.

[0030] In an embodiment, the step of predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data is performed by a bandwidth prediction model, wherein the bandwidth prediction model is a bidirectional gated recurrent unit network based on attention mechanism, i.e., a bidirectional GRU neural network.

[0031] In yet another embodiment, the training step of the bandwidth prediction model can include: collecting sample index data of each performance index of the storage cluster at different sample time points and corresponding bandwidths, wherein the bandwidth is used as the label data of the corresponding sample index data. At least one performance index group is formed by using at least one performance index, and different performance index groups include the same or different number of performance indexes, and there is at least one different performance index between each performance index group. For each performance index group, the sample index data corresponding to the performance index group is used as a data set, and the data set is divided into a training set and a test set according to time sequence, wherein the data amount of the training set is greater than that of the test set. The bandwidth prediction model is trained by using the training set corresponding to each performance index group, and the trained bandwidth prediction model is tested by using the test set corresponding to each performance index group to obtain a test result. If the test result does not meet the test requirement, the bandwidth prediction model is repeatedly trained by using the original training set or a new training set until the test result meets the test requirement.

[0032] In a specific embodiment, the sample index data of each performance index of the storage cluster at different sample time points and the corresponding bandwidths can be subjected to data cleaning and dimensionless processing. The specific processing manner can be referred to the related description in step S11.

[0033] In yet another specific embodiment, the performance indicators include total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster traffic, network latency, at least one of the performance indicators is used to form at least one performance indicator group, and sample indicator data corresponding to the performance indicator group is taken as a data set. For example, total bandwidth, total write bandwidth, total read bandwidth, and total data recovery bandwidth are used to form a performance indicator group, and the performance indicator group is taken as a data set.

[0034] In yet another specific embodiment, the data set can be divided into an 80% training set and a 20% test set in chronological order. The proportions of the training set and the test set in the data set can be adjusted according to actual needs, and are not limited herein. Taking the data set divided into an 80% training set and a 20% test set in chronological order as an example, all sample indicator data in the data set is divided into 20 segments in chronological order, 4 are randomly selected from 1 / 20, 2 / 20, …, 19 / 20, and 20 / 20, sample indicator data corresponding to the 4 time intervals is taken as a test set, and sample indicator data corresponding to the remaining 16 time intervals is taken as a training set. Of course, the 20 segments can be further divided in each time interval, 4 segments of sample indicator data are randomly selected as a test set, and sample indicator data corresponding to the remaining 16 time intervals is taken as a training set. The selection can be made according to actual needs, and is not limited herein.

[0035] In yet another specific embodiment, before training the bandwidth prediction model, the network parameters of the bandwidth prediction model can be initialized. In the case that the test result does not meet the test requirement, the bandwidth prediction model is repeatedly trained using the original training set or a new training set, the network parameters of the bandwidth prediction model are fine-tuned until the test result meets the test requirement. The network parameters can include the number of hidden layers, the number of neurons, the learning rate, the iteration batch, the batch size, the setting of the loss function, and the selection of the optimizer.

[0036] In combination Figure 2 For illustration, Figure 2 is a training process schematic diagram of the bandwidth prediction model of the data recovery method provided by the present application, wherein the bandwidth prediction model is a bidirectional gated recurrent unit network based on attention mechanism. As shown in Figure 2 The training of the bandwidth prediction model provided by the present embodiment can include the following steps:

[0037] Step S21: Collecting sample indicator data of each performance indicator of the storage cluster at different sample moments and corresponding bandwidth to obtain a data set. Step S22: Data cleaning and dimensionless processing are performed on the data set to obtain a processed data set. Step S23: The processed data set is divided into a training set and a test set.

[0038] Step S24: initialize network parameters. After performing steps S23 and S24, perform step S25: train the bidirectional gated recurrent unit network based on attention mechanism with the training set until convergence. Step S26: test the trained bidirectional gated recurrent unit network with the test set. Step S27: determine whether the test result meets the test requirement. In the case of yes in step S27, perform step S28: the bidirectional gated recurrent unit network training is completed, that is, the bandwidth prediction model training is completed. In the case of no in step S27, perform step S29: fine-tune the network parameters. Perform step S25.

[0039] In yet another embodiment, based on the first index data, the step of predicting the predicted bandwidth data of the storage cluster in the next time period is performed by a bandwidth prediction model. The bandwidth prediction model is a bidirectional gated recurrent unit network based on attention mechanism. Wherein, the training steps of the bandwidth prediction model can refer to the related description in step S12, which will not be repeated here.

[0040] Step S13: based on the predicted bandwidth data, determine the bandwidth allocation amount for data recovery in the next time period, so as to allocate the bandwidth resources of the storage cluster in the next time period according to the bandwidth allocation amount for data recovery.

[0041] In an embodiment, considering that the write performance of the storage cluster should also be guaranteed while data recovery is being performed, the bandwidth allocation amount is less than or equal to the difference between the total available bandwidth resource amount of the storage cluster in the next time period and the minimum write bandwidth resource amount, and the minimum write bandwidth resource amount is the minimum bandwidth resource amount required for normal write business of the storage cluster in the next time period.

[0042] For example, the total available bandwidth resource of the storage cluster in the next time period is 100 Gbps, and the minimum bandwidth resource amount required to maintain normal write business is 60 Gbps. Then, when performing data recovery, the bandwidth allocation amount is controlled within 40 Gbps (i.e. 100 Gbps-60 Gbps=40 Gbps). In this way, even if data recovery is being performed, the write business of the storage cluster can be guaranteed.

[0043] In yet another embodiment, the predicted bandwidth data is used to determine the bandwidth usage of the storage cluster in the next time period, and the bandwidth usage includes a bandwidth resource usage trend. In response to the bandwidth resource usage trend being a peak trend, the bandwidth allocation amount for data recovery in the next time period is determined within a range below a peak bandwidth allocation upper limit amount. In response to the bandwidth resource usage trend being a valley trend, the bandwidth allocation amount for data recovery in the next time period is determined within a range below a valley bandwidth allocation upper limit amount. Wherein, the peak bandwidth allocation upper limit amount is less than the valley bandwidth allocation upper limit amount.

[0044] For example, the peak trend can be understood as an upward trend in which the predicted bandwidth data is expected to increase, which can represent that the storage cluster is short of bandwidth resources in the next time period. The valley trend can be understood as a downward trend in which the predicted bandwidth data is expected to decrease, which can represent that the storage cluster is relatively abundant in bandwidth resources in the next time period. The bandwidth allocation amount for data recovery in the next time period can be determined according to the predicted bandwidth resource usage trend.

[0045] In an embodiment, considering that the peak value corresponding to the peak trend can not always be greater than the valley value corresponding to the valley trend and other complex and changeable actual situations, a current peak / valley degree representation value can be introduced, which is intended to reflect the performance degree of the peak / valley phenomenon. That is, the bandwidth usage can further include the current peak / valley degree representation value, the bandwidth allocation amount corresponding to the peak trend is negatively correlated with the current peak degree representation value, and the bandwidth allocation amount corresponding to the valley trend is positively correlated with the current valley degree representation value.

[0046] For example, the current peak / valley degree representation value is determined according to the evaluation of the intensity and duration of the peak / valley according to the predicted bandwidth data. Specifically, a median boundary (such as a mathematical expectation boundary) of the peak and the valley is set, and the greater the deviation from the median boundary and the longer the duration, the more obvious the peak / valley phenomenon, that is, the greater the current peak / valley degree representation value. In the case of a greater peak degree representation value, the storage cluster is represented as having less bandwidth resources in the next time period, and the corresponding bandwidth allocation amount is less. Similarly, in the case of a greater valley degree representation value, the storage cluster is represented as having more bandwidth resources in the next time period, and the corresponding bandwidth allocation amount is more.

[0047] In an embodiment, the predicted bandwidth data and the bandwidth change of the actual bandwidth data of the storage cluster before the next time period can be analyzed to determine the bandwidth resource usage trend of the storage cluster in the next time period. The specific time period of the actual bandwidth data of the storage cluster before the next time period can be adjusted according to actual needs, for example, the actual bandwidth data of the storage cluster within 1 day before the next time period. Of course, the bandwidth change of the predicted bandwidth data can also be analyzed to determine the bandwidth resource usage trend of the storage cluster in the next time period, which is not limited here.

[0048] In yet another embodiment, in the case that the bandwidth resource usage trend is a peak trend, in response to the level of the data to be recovered being an emergency recovery level, the bandwidth allocation amount of the data to be recovered in the next time period is determined as a first bandwidth resource amount. In response to the level of the data to be recovered not being an emergency recovery level, the bandwidth allocation amount of the data to be recovered in the next time period is determined as a second bandwidth resource amount. In the case that the bandwidth resource usage trend is a trough trend, in response to the level of the data to be recovered being an emergency recovery level, the bandwidth allocation amount of the data to be recovered in the next time period is determined as a third bandwidth resource amount. In response to the level of the data to be recovered not being an emergency recovery level, the bandwidth allocation amount of the data to be recovered in the next time period is determined as a fourth bandwidth resource amount.

[0049] In an embodiment, the first bandwidth resource amount and the second bandwidth resource amount satisfy at least one of the following conditions: the first bandwidth resource amount is greater than 0 and the second bandwidth resource amount is greater than or equal to 0, and the first bandwidth resource amount is greater than the second bandwidth resource amount.

[0050] For example, in the case that the bandwidth resource usage trend is a peak trend, it indicates that the bandwidth resources of the storage cluster are insufficient, and the bandwidth allocation amount of the data to be recovered should be small to ensure the normal writing of the remaining data in the storage cluster. However, in the case that the level of the data to be recovered is an emergency recovery level, the first bandwidth resource amount allocated is greater than 0 to ensure that the data recovery work can proceed smoothly. Conversely, in the case that the level of the data to be recovered is not an emergency recovery level, the bandwidth demand of the write operation should be prioritized, i.e., the second bandwidth resource amount should be greater than 0, or even the second bandwidth resource amount can be equal to 0 to ensure the normal execution of the write operation. Of course, in determining the bandwidth allocation amount, the first bandwidth resource amount corresponding to the data to be recovered of the emergency recovery level can be greater than the second bandwidth resource amount corresponding to the data to be recovered of the non-emergency recovery level.

[0051] In yet another embodiment, the third bandwidth resource amount and the fourth bandwidth resource amount satisfy at least one of the following conditions: the third bandwidth resource amount and the fourth bandwidth resource amount are greater than 0, the third bandwidth resource amount is greater than the first bandwidth resource amount, and the third bandwidth resource amount is greater than the fourth bandwidth resource amount.

[0052] For example, in the case of a bandwidth resource usage trend being a trough trend, it indicates that the bandwidth resources of the storage cluster are sufficient, and the bandwidth allocation amount of the data to be recovered can be larger. In the case of the level of the data to be recovered being an emergency recovery level, the third bandwidth resource amount is greater than 0, to ensure that the data recovery work can proceed smoothly. In the case of the level of the data to be recovered not being an emergency recovery level, the bandwidth demand of the write operation should be prioritized, and since the bandwidth resources of the storage cluster are sufficient, the fourth bandwidth resource amount can be greater than 0. In some scenarios, the bandwidth allocation amount determined when the bandwidth resource usage trend is a trough trend can be greater than the bandwidth allocation amount when the bandwidth resource usage trend is a peak trend, that is, the third bandwidth resource amount can be greater than the first bandwidth resource amount. In yet other scenarios, the bandwidth allocation amount for the emergency recovery level when the bandwidth resource usage trend is a trough trend can be greater than the bandwidth allocation amount that does not belong to the emergency recovery level, that is, the third bandwidth resource amount is greater than the fourth bandwidth resource amount.

[0053] In yet another specific implementation, the first bandwidth resource amount and the second bandwidth resource amount satisfy at least one of the following conditions: the first bandwidth resource amount is greater than 0, and the second bandwidth resource amount is greater than or equal to 0, the first bandwidth resource amount is greater than the second bandwidth resource amount. The third bandwidth resource amount and the fourth bandwidth resource amount satisfy at least one of the following conditions: the third bandwidth resource amount and the fourth bandwidth resource amount are greater than 0, the third bandwidth resource amount is greater than the first bandwidth resource amount, and the third bandwidth resource amount is greater than the fourth bandwidth resource amount.

[0054] In yet another implementation, different data to be recovered are divided into different recovery levels based on data damage conditions, wherein the higher the recovery level of the data to be recovered, the higher the execution priority of the data recovery.

[0055] For example, according to the damage degree of the data to be recovered, a plurality of priority queues of different recovery levels are set, and the recovery level of the data to be recovered is proportional to the execution priority thereof, that is, the higher the recovery level, the higher the execution priority. The priority queues are, for example, queue(N)>queue(N-1)>…queue(1), and the execution priority of queue(N) is greater than that of queue(N-1), queue(N) indicating that the data to be recovered represented in the queue has been damaged N times, and the recovery level of the data to be recovered is the highest. The recovery level corresponding to queue(N) is taken as an emergency recovery level, and the remaining recovery levels do not belong to the emergency recovery level. For example, the data to be recovered is in an M+N erasure code storage manner or an N+1 copy storage manner, so that a maximum of N pieces of encoded data can be damaged, and further damage will cause permanent damage and cannot be recovered. When the damage number of the data to be recovered changes from N-1 to N, the data to be recovered is dequeued from queue(N-1) and enqueued to queue(N), and at this time, the recovery level of the data to be recovered is the highest, which is the emergency recovery level.

[0056] In yet another embodiment, different to-be-restored data are classified into different recovery levels based on data damage conditions, wherein the higher the recovery level of the to-be-restored data is, the more the corresponding bandwidth allocation amount is.

[0057] In yet another embodiment, different to-be-restored data are classified into different recovery levels based on data damage conditions, wherein the higher the recovery level of the to-be-restored data is, the higher the execution priority of data recovery is, and the more the corresponding bandwidth allocation amount is.

[0058] In yet another embodiment, in response to the recovery level of the to-be-restored data being an emergency recovery level, the execution priority of the to-be-restored data in the next time period is determined to be higher than or equal to the execution priority of the write data. In response to the recovery level of the to-be-restored data not being an emergency recovery level, the execution priority of the to-be-restored data in the next time period is determined to be equal to or lower than the execution priority of the write data.

[0059] For example, in response to the recovery level of the to-be-restored data not being an emergency recovery level, it is indicated that the to-be-restored data does not need to be recovered urgently, and therefore the write data can be executed first. Of course, it is also possible to execute the to-be-restored data and the write data at the same time.

[0060] In yet another embodiment, every preset time period, the data set (validation set and training set) of the bandwidth prediction model is obtained from the data corresponding to the performance indicators of the new storage cluster generated in the preset time period, and the bandwidth prediction model is further trained and calibrated, so that the training effect of the bandwidth prediction model forms a positive feedback mechanism, that is, the bandwidth prediction model is in continuous feedback adjustment and training. For example, every week, the data set of the bandwidth prediction model is obtained from the data corresponding to the performance indicators of the storage cluster, so as to further train the bandwidth prediction model. The training steps of the further training of the bandwidth prediction model can refer to the related description of the training steps of the bandwidth prediction model in step S12, and will not be described here.

[0061] In yet another embodiment, the bandwidth resources in the storage cluster can be monitored in real time, and the bandwidth allocation amount of data recovery in the next time period is adjusted to prevent abnormal situations and large prediction deviations from causing abnormal cluster write services. If the bandwidth resource consumption abnormally increases, the bandwidth allocation amount needs to be controlled, and if the prediction deviation is small and the bandwidth resources are sufficient, the bandwidth allocation amount of the to-be-restored data can be increased.

[0062] In combination Figure 3 For example, Figure 3 is a part of the flowchart of an embodiment of the data recovery method provided in the present application. As shown in Figure 3 , the data recovery method provided in the present embodiment can include the following steps:

[0063] Step S31: obtaining first index data of the storage cluster in a current time period. Step S32: predicting, by using a prediction bandwidth model, prediction bandwidth data of the storage cluster in a next time period based on the first index data. Step S33: determining whether the prediction bandwidth data is a peak trend.

[0064] Step S34: initializing a priority queue provided with a plurality of different recovery levels. Step S35: updating the priority queue in real time according to a data damage condition. Step S36: scheduling the data to be recovered according to the execution priority of the recovery levels in the priority queue. Step S37: determining whether the recovery level of the data to be recovered is an emergency recovery level.

[0065] In a case where the determination result of step S33 is yes and the determination result of step S37 is yes, step S38 is performed to determine that the bandwidth allocation amount of the data to be recovered in the next time period is a first bandwidth resource amount, and the first bandwidth resource amount is greater than 0. In a case where the determination result of step S33 is yes and the determination result of step S37 is no, step S39 is performed to determine that the bandwidth allocation amount of the data to be recovered in the next time period is a second bandwidth resource amount, and the second bandwidth resource amount is greater than or equal to 0. In a case where the determination result of step S33 is no and the determination result of step S37 is yes, step S310 is performed to determine that the bandwidth allocation amount of the data to be recovered in the next time period is a third bandwidth resource amount, and the third bandwidth resource amount is greater than 0. In a case where the determination result of step S33 is no and the determination result of step S37 is no, step S311 is performed to determine a fourth bandwidth resource amount of the data recovery in the next time period within a range below a lower limit of the trough bandwidth allocation, and the fourth bandwidth resource amount is greater than 0.

[0066] Please refer to Figure 4 , Figure 4 is a flowchart of an embodiment of the data recovery device. The data recovery device 400 comprises a first index data acquisition module 410, a prediction bandwidth data acquisition module 420, and a bandwidth allocation amount acquisition module 430. The first index data acquisition module 410 is configured to acquire first index data of the storage cluster in a current time period, wherein the first index data comprises data of at least one performance index of the storage cluster. The prediction bandwidth data acquisition module 420 is configured to predict prediction bandwidth data of the storage cluster in a next time period based on the first index data, wherein the prediction bandwidth data is used to represent the bandwidth resource usage of the storage cluster. The bandwidth allocation amount acquisition module 430 is configured to determine a bandwidth allocation amount of data recovery in the next time period based on the prediction bandwidth data, so as to allocate the bandwidth resource of the storage cluster in the next time period according to the bandwidth allocation amount for data recovery.

[0067] In some embodiments, the bandwidth allocation amount is less than or equal to a difference between a total available bandwidth resource amount of the storage cluster in the next time period and a minimum write bandwidth resource amount, the minimum write bandwidth resource amount being a minimum bandwidth resource amount required by the storage cluster for normal write service in the next time period.

[0068] In some embodiments, the bandwidth allocation amount obtaining module 430, in performing the determining of the bandwidth allocation amount for data recovery in the next time period based on the predicted bandwidth data, comprises: determining, by using the predicted bandwidth data, a bandwidth usage of the storage cluster in the next time period, the bandwidth usage comprising a bandwidth resource usage trend. In response to the bandwidth resource usage trend being a peak trend, the bandwidth allocation amount for data recovery in the next time period is determined within a range below a peak bandwidth allocation upper limit amount. And / or, in response to the bandwidth resource usage trend being a valley trend, the bandwidth allocation amount for data recovery in the next time period is determined within a range below a valley bandwidth allocation upper limit amount. Wherein, the peak bandwidth allocation upper limit amount is less than the valley bandwidth allocation upper limit amount.

[0069] In some embodiments, the bandwidth usage further comprises a current peak / valley degree representation value, the bandwidth allocation amount corresponding to the peak trend is negatively correlated with the current peak degree representation value, and the bandwidth allocation amount corresponding to the valley trend is positively correlated with the current valley degree representation value.

[0070] In some embodiments, the bandwidth allocation amount obtaining module 430, in performing the determining of the bandwidth usage of the storage cluster in the next time period by using the predicted bandwidth data, comprises: analyzing the predicted bandwidth data and a bandwidth change of actual bandwidth data of the storage cluster before the next time period to determine a bandwidth resource usage trend of the storage cluster in the next time period.

[0071] In some embodiments, the bandwidth usage further comprises a current peak / valley degree representation value, the bandwidth allocation amount corresponding to the peak trend is negatively correlated with the current peak degree representation value, and the bandwidth allocation amount corresponding to the valley trend is positively correlated with the current valley degree representation value. The bandwidth allocation amount obtaining module 430, in performing the determining of the bandwidth usage of the storage cluster in the next time period by using the predicted bandwidth data, comprises: analyzing the predicted bandwidth data and a bandwidth change of actual bandwidth data of the storage cluster before the next time period to determine a bandwidth resource usage trend of the storage cluster in the next time period.

[0072] In some embodiments, the bandwidth allocation amount obtaining module 430, in the specific implementation of the range below the upper limit of the peak bandwidth allocation, determines the bandwidth allocation amount of the data recovery in the next time period, including: in response to the level of the data to be recovered being an emergency recovery level, determining the bandwidth allocation amount of the data to be recovered in the next time period as a first bandwidth resource amount. In response to the level of the data to be recovered not being an emergency recovery level, determining the bandwidth allocation amount of the data to be recovered in the next time period as a second bandwidth resource amount. The bandwidth allocation amount obtaining module 430, in the specific implementation of the range below the upper limit of the valley bandwidth allocation, determines the bandwidth allocation amount of the data recovery in the next time period, including: in response to the level of the data to be recovered being an emergency recovery level, determining the bandwidth allocation amount of the data to be recovered in the next time period as a third bandwidth resource amount. In response to the level of the data to be recovered not being an emergency recovery level, determining the bandwidth allocation amount of the data to be recovered in the next time period as a fourth bandwidth resource amount.

[0073] In some embodiments, the first bandwidth resource amount and the second bandwidth resource amount satisfy at least one of the following conditions: the first bandwidth resource amount is greater than 0, and the second bandwidth resource amount is greater than or equal to 0, and the first bandwidth resource amount is greater than the second bandwidth resource amount.

[0074] In some embodiments, the third bandwidth resource amount and the fourth bandwidth resource amount satisfy at least one of the following conditions: the third bandwidth resource amount and the fourth bandwidth resource amount are greater than 0, the third bandwidth resource amount is greater than the first bandwidth resource amount, and the third bandwidth resource amount is greater than the fourth bandwidth resource amount.

[0075] In some embodiments, the first bandwidth resource amount and the second bandwidth resource amount satisfy at least one of the following conditions: the first bandwidth resource amount is greater than 0, and the second bandwidth resource amount is greater than or equal to 0, and the first bandwidth resource amount is greater than the second bandwidth resource amount. The third bandwidth resource amount and the fourth bandwidth resource amount satisfy at least one of the following conditions: the third bandwidth resource amount and the fourth bandwidth resource amount are greater than 0, the third bandwidth resource amount is greater than the first bandwidth resource amount, and the third bandwidth resource amount is greater than the fourth bandwidth resource amount.

[0076] In some embodiments, different data to be recovered is classified into different recovery levels based on data damage conditions, wherein the higher the recovery level of the data to be recovered, the higher the execution priority of the data recovery.

[0077] In some embodiments, different data to be recovered is classified into different recovery levels based on data damage conditions, wherein the higher the recovery level of the data to be recovered, the more the corresponding bandwidth allocation amount.

[0078] In some embodiments, different data to be recovered is classified into different recovery levels based on data damage conditions, wherein the higher the recovery level of the data to be recovered, the higher the execution priority of the data recovery, and the more the corresponding bandwidth allocation amount.

[0079] In some embodiments, the data recovery device 400 further comprises an execution priority determination module, which is configured to determine, in response to the level of the to-be-recovered data being the emergency recovery level, an execution priority of the to-be-recovered data to be recovered in the next time period to be higher than or equal to the execution priority of the write data. In response to the level of the to-be-recovered data not being the emergency recovery level, the execution priority of the to-be-recovered data to be recovered in the next time period is determined to be equal to or lower than the execution priority of the write data.

[0080] In some embodiments, the step of predicting the predicted bandwidth data of the storage cluster in the next time period based on the first index data is performed by a bandwidth prediction model, wherein the bandwidth prediction model is a bidirectional gated recurrent unit network based on an attention mechanism.

[0081] In some embodiments, the training step of the bandwidth prediction model comprises: collecting sample index data of each performance index of the storage cluster at different sample time points and corresponding bandwidths, wherein the bandwidth is used as labeled data corresponding to the sample index data. At least one performance index group is formed by using at least one performance index, different performance index groups include the same or different number of performance indexes, and there is at least one different performance index between each performance index group. For each performance index group, the sample index data corresponding to the performance index group is used as a data set, and the data set is divided into a training set and a test set according to time sequence, wherein the data amount of the training set is greater than that of the test set. The bandwidth prediction model is trained by using the training set corresponding to each performance index group, and the trained bandwidth prediction model is tested by using the test set corresponding to each performance index group to obtain a test result. If the test result does not meet the test requirement, the bandwidth prediction model is repeatedly trained by using the original training set or a new training set until the test result meets the test requirement.

[0082] In some embodiments, the at least one performance index comprises at least one of total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster traffic, and network delay.

[0083] In some embodiments, the first index data acquisition module 410 comprises: for each performance index in the at least one performance index, collecting data corresponding to the performance index at different time points in the current time period to obtain the first index data, wherein the first index data comprises a multi-dimensional vector corresponding to each time point, and the multi-dimensional vector comprises data of the at least one performance index at the time point.

[0084] In some embodiments, before the bandwidth allocation amount obtaining module 430 performs prediction of the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the method further comprises data cleaning of the first index data.

[0085] In some embodiments, before the bandwidth allocation amount obtaining module 430 performs prediction of the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the method further comprises dimensionless processing of different dimensional data in the first index data.

[0086] In some embodiments, before the bandwidth allocation amount obtaining module 430 performs prediction of the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the method further comprises data cleaning of the first index data and dimensionless processing of different dimensional data in the first index data.

[0087] In some embodiments, the at least one performance index comprises at least one of total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster service, and network delay. The first index data obtaining module 410, when performing obtaining of the first index data of the storage cluster in the current time period, comprises: for each performance index in the at least one performance index, collecting data corresponding to different time points in the current time period of the performance index to obtain the first index data, wherein the first index data comprises a multi-dimensional vector corresponding to each time point, and the multi-dimensional vector comprises data of the at least one performance index at the corresponding time point. Before the bandwidth allocation amount obtaining module 430 performs prediction of the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the method further comprises data cleaning of the first index data.

[0088] In some embodiments, the at least one performance index comprises at least one of total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster service, and network delay. The first index data obtaining module 410, when performing obtaining of the first index data of the storage cluster in the current time period, comprises: for each performance index in the at least one performance index, collecting data corresponding to different time points in the current time period of the performance index to obtain the first index data, wherein the first index data comprises a multi-dimensional vector corresponding to each time point, and the multi-dimensional vector comprises data of the at least one performance index at the corresponding time point. Before the bandwidth allocation amount obtaining module 430 performs prediction of the predicted bandwidth data of the storage cluster in the next time period based on the first index data, the method further comprises dimensionless processing of different dimensional data in the first index data.

[0089] In some embodiments, the at least one performance indicator includes at least one of total bandwidth, total write bandwidth, total read bandwidth, total data recovery bandwidth, CPU usage, memory usage, disk input / output, total IOPS of cluster traffic, network latency. The first indicator data acquisition module 410, in performing the acquiring of the first indicator data of the storage cluster in the current time period, includes, for each performance indicator in the at least one performance indicator, collecting data corresponding to the performance indicator at different time instants in the current time period to obtain the first indicator data, wherein the first indicator data includes a plurality of multi-dimensional vectors corresponding to the different time instants, and each multi-dimensional vector includes data of the at least one performance indicator at a time instant. Before the bandwidth allocation amount acquisition module 430 performs the predicting of the predicted bandwidth data of the storage cluster in the next time period based on the first indicator data, the first indicator data is further subjected to data cleaning, and different dimension data in the first indicator data is subjected to dimensionless processing.

[0090] Referring to Figure 5 , Figure 5 is a schematic diagram of a framework of an embodiment of the electronic device. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is configured to execute program instructions stored in the memory 51 to implement the steps in any of the above data recovery method embodiments. In a specific implementation scenario, the electronic device 50 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 50 can also include a notebook computer, a tablet computer, and the like, without limitation.

[0091] Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps in any of the above data recovery method embodiments. The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 can be an integrated circuit chip having a processing capability of signals. The processor 52 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 52 can be implemented by an integrated circuit chip together.

[0092] Referring to Figure 6 , Figure 6FIG. 1 is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program instructions 61 capable of being executed by a processor, and the program instructions 61 are used to implement the steps in any of the data recovery method embodiments described above.

[0093] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0094] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0095] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the apparatus implementation described above is only schematic, and the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0096] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0097] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that makes a contribution to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A data recovery method, characterized in that, The method includes: Obtain the first indicator data of the storage cluster in the current time period, wherein the first indicator data includes data of at least one performance indicator of the storage cluster; Based on the first indicator data, the predicted bandwidth data of the storage cluster in the next time period is predicted, wherein the predicted bandwidth data is used to characterize the bandwidth resource usage of the storage cluster. Based on the predicted bandwidth data, determine the bandwidth allocation amount for data recovery in the next time period, so as to allocate the bandwidth resources of the storage cluster for data recovery in the next time period according to the bandwidth allocation amount; The step of determining the bandwidth allocation for data recovery in the next time period based on the predicted bandwidth data includes: Using the predicted bandwidth data, the bandwidth usage of the storage cluster in the next time period is determined, and the bandwidth usage includes bandwidth resource usage trends; In response to the fact that the bandwidth resource usage trend is a peak trend and the level of the data to be recovered is an emergency recovery level, the bandwidth allocation of the data to be recovered in the next time period is determined as the first bandwidth resource amount; In response to the fact that the bandwidth resource usage trend is a peak trend and the level of the data to be restored is not an emergency recovery level, the bandwidth allocation amount of the data to be restored in the next time period is determined as the second bandwidth resource amount. The first bandwidth resource amount and the second bandwidth resource amount satisfy at least one of the following conditions: the first bandwidth resource amount is greater than 0 and the second bandwidth resource amount is greater than or equal to 0, and the first bandwidth resource amount is greater than the second bandwidth resource amount.

2. The method according to claim 1, characterized in that, The bandwidth allocation is less than or equal to the difference between the total available bandwidth resources of the storage cluster in the next time period and the minimum write bandwidth resources, where the minimum write bandwidth resources are the minimum bandwidth resources required for the storage cluster to perform normal write operations in the next time period.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In response to the fact that the bandwidth resource usage trend is a trough trend, within a range below the upper limit of the trough bandwidth allocation, the bandwidth allocation for data recovery in the next time period is determined; Wherein, the upper limit of the peak bandwidth allocation is less than the upper limit of the trough bandwidth allocation.

4. The method according to claim 3, characterized in that, The bandwidth usage also includes the current peak / valley level characterization value. The bandwidth allocation corresponding to the peak trend is negatively correlated with the current peak level characterization value, and the bandwidth allocation corresponding to the valley trend is positively correlated with the current valley level characterization value. And / or, determining the bandwidth usage of the storage cluster in the next time period using the predicted bandwidth data includes: Analyze the bandwidth changes of the predicted bandwidth data and the actual bandwidth data of the storage cluster before the next time period to determine the bandwidth resource usage trend of the storage cluster in the next time period.

5. The method according to claim 3, characterized in that, Determining the bandwidth allocation for data recovery within the next time period, within the range below the upper limit of the trough bandwidth allocation, includes: In response to the fact that the level of the data to be recovered is the emergency recovery level, the bandwidth allocation of the data to be recovered in the next time period is determined to be the third bandwidth resource amount; In response to the fact that the level of the data to be recovered is not an emergency recovery level, the bandwidth allocation of the data to be recovered in the next time period is determined to be the fourth bandwidth resource amount.

6. The method according to claim 5, characterized in that, The third bandwidth resource quantity and the fourth bandwidth resource quantity satisfy at least one of the following conditions: the third bandwidth resource quantity and the fourth bandwidth resource quantity are greater than 0, the third bandwidth resource quantity is greater than the first bandwidth resource quantity, and the third bandwidth resource quantity is greater than the fourth bandwidth resource quantity.

7. The method according to claim 1 or 2, characterized in that, Different data to be recovered are classified into different recovery levels based on the degree of data corruption. The higher the recovery level of the data to be recovered, the higher the execution priority of data recovery and / or the greater the corresponding bandwidth allocation.

8. The method according to claim 1, characterized in that, The step of predicting the bandwidth data of the storage cluster in the next time period based on the first indicator data is executed by the bandwidth prediction model, wherein, The bandwidth prediction model is a bidirectional gated recurrent unit network based on an attention mechanism; and / or, the training steps of the bandwidth prediction model include: Collect sample index data of various performance indicators of the storage cluster at different sample times and the corresponding bandwidth, wherein the bandwidth is used as the labeled data of the corresponding sample index data. Each performance index group is formed by using at least one of the aforementioned performance indices. Different performance index groups may include the same or different number of performance indices, and there is at least one different performance index among the various performance index groups. For each group of performance metrics, the sample metrics data corresponding to the performance metrics group are taken as a dataset, and the dataset is divided into a training set and a test set according to the time sequence, wherein the data volume of the training set is greater than the data volume of the test set. The bandwidth prediction model is trained using the training set corresponding to each group of performance indicators, and the trained bandwidth prediction model is tested using the test set corresponding to each group of performance indicators to obtain test results. If the test results do not meet the test requirements, the bandwidth prediction model is trained again using the original training set or a new training set until the test results meet the test requirements.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to implement the data recovery method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the data recovery method as described in any one of claims 1-8.

Images