A cloud-edge collaborative data fault-tolerant method and system

Abstract

The present invention provides a cloud-edge collaborative data fault-tolerant method, including: constructing and training two fault prediction models with different precisions, the two fault prediction models are located in the Internet of Things terminal and the cloud, respectively; collecting hard disk SMART data and adding time series features and trend features to it, inputting the SMART data into the fault prediction model located in the Internet of Things terminal for preliminary fault prediction; when the preliminary prediction result is that the hard disk is on the verge of failure, the SMART data carrying the time series features and trend features are sent to the fault prediction model located in the cloud for secondary fault prediction; when the secondary prediction result is that the hard disk is on the verge of failure, the hard disk is repaired before the hard disk fails by performing data migration and reconstruction operations in parallel. Beneficial effects of the present invention: Through the cloud-edge coordinated data fault-tolerant method, the technicians are notified in advance to take measures, and the data in these hard disks are actively repaired before the failure actually occurs, thereby ensuring data security and reducing maintenance costs.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular, relates to a cloud-edge collaborative data fault-tolerant method and system. Background Art

[0002] With the rapid development of the Internet of Things, terminal devices have generated a large amount of data. These data need to be transmitted and processed in a timely manner in order to make real-time decisions or conduct remote monitoring. For the Internet of Things platform, it is very important to ensure the reliability and smooth operation of terminal data. However, today, when hard disks are widely used in the Internet of Things platform, hard disks have become the most prone to failure in Internet of Things terminal devices. It is estimated that in application environments represented by the Internet of Things, 78% of hardware replacements are caused by hard disk failures. If the hard disk fails, it will not only cause the Internet of Things terminal service to be unavailable, but more seriously, it may cause permanent data loss, thereby causing significant losses to organizations or individuals. Due to the particularity of the operating environment of Internet of Things devices, many Internet of Things terminals need to be used outdoors, even in extremely harsh temperatures and humidity, and often experience wind and rain. The probability of hard disk failure is significantly higher than that in the data center environment. The phenomenon of data loss in Internet of Things terminals due to hard disk failure is extremely frequent. Therefore, how to improve the data reliability of Internet of Things devices has become a very important task.

[0003] Traditional IoT terminals do not take any reliability guarantees, do not store any redundant information, and aim to be extremely low-cost. In this case, once a hard disk fails, the valuable data collected by the IoT terminal will be permanently lost. Some research works have tried to use multiple copies in IoT terminals to improve reliability, but data consistency faces great challenges. All data collected by IoT terminals must be redundantly written to other devices, which will lead to a large amount of data transmission, increase network overhead and system load. The number of IoT terminal devices is large and cost control requirements are strict. The multiple copies method is unacceptable to most IoT applications.

[0004] Erasure coding is a data encoding technology that combines low storage overhead with high fault tolerance. Erasure coding seeks a balance between storage efficiency and fault tolerance through two parameters, k and m, and uses operations such as encoding (i.e. calculating additional redundant data) and decoding (i.e. repairing the original data) to achieve mutual conversion between original data and redundant data. Using erasure coding technology to transform and upgrade IoT terminals is a feasible method, but in order to avoid frequent data transmission, actual IoT terminals often calculate check blocks periodically and save them on hard disks near the terminals.

[0005] The advantage of this solution is that it can restore data to the last time point after a terminal failure, with less interference to performance and acceptable cost. However, it will cause the latest data to be lost, and the latest data in the IoT scenario is often the most important. After a hard disk failure, the reconstruction process requires additional hard disk read and write and computing operations, so system degradation will occur during the reconstruction of the failed disk. Data repair will not begin until the hard disk failure is detected, and the front-end business needs to be suspended. This is a big threat to IoT applications or IoT devices that have strict requirements on high availability and low latency. Summary of the invention

[0006] In view of this, the present invention aims to propose a cloud-edge collaborative data fault-tolerant method and system, in order to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The first aspect of the present invention proposes a cloud-edge collaborative data fault tolerance method, comprising:

[0009] Build and train two fault prediction models with different accuracies, one in the IoT terminal and the other in the cloud.

[0010] Collect hard disk SMART data and add time series features and trend features to it, and input the SMART data into the fault prediction model located in the IoT terminal for preliminary fault prediction;

[0011] When the initial prediction result shows that the hard disk is on the verge of failure, the SMART data with time series characteristics and trend characteristics will be sent to the failure prediction model in the cloud for secondary failure prediction;

[0012] When the secondary prediction result is that the hard disk is on the verge of failure, the hard disk is repaired before the hard disk fails by performing data migration and reconstruction operations in parallel.

[0013] Furthermore, the IoT terminal continuously collects business data and stores it locally, calculates redundant data corresponding to the business data by erasure coding, and stores it in the cloud;

[0014] When the initial prediction result is that the hard disk is on the verge of failure, the hard disk SMART data is sent to the cloud. If the result of the secondary failure prediction performed on the cloud is still that the hard disk is on the verge of failure, the locally stored business data and the redundant data in the cloud are used to perform data migration and reconstruction operations.

[0015] Furthermore, the process of training two fault prediction models with different precisions includes:

[0016] The hard disk failure model is trained by combining the gradient boosting decision tree model with the training set, and the model is tuned through grid search and cross-validation to obtain a lightweight fault prediction model and a full-scale fault prediction model.

[0017] The lightweight fault prediction model is distributed on each IoT terminal, and the full-scale fault prediction model is uploaded to the cloud.

[0018] Furthermore, data stability is maintained through timing features and trend features, wherein the timing features are used to evaluate the time series complexity of the hard disk SMART data, and the trend features are used to reflect the changing trend of the hard disk SMART data; wherein the timing features are introduced through a sliding window method.

[0019] Furthermore, the process of performing the data migration and reconstruction operation includes:

[0020] Select healthy hard disks with free capacity that meets the needs of migration and reconstruction, set a threshold for the repair ratio, and divide the data blocks to be repaired into two parts: migration and reconstruction according to the threshold;

[0021] By adjusting the repair ratio threshold, the execution time of data migration is close to that of data reconstruction.

[0022] Data migration is performed on the data blocks to be repaired in the migration part, and data reconstruction is performed on the data blocks to be repaired in the reconstruction part.

[0023] Furthermore, the data migration uses a direct copy method to perform data repair, copying the data of the hard disk on the verge of failure to the target healthy disk;

[0024] The data reconstruction uses an erasure code method to repair the data, and uses the reconstructed hard disk group to perform encoding and decoding calculations to obtain corresponding data block content and store it in a target healthy disk.

[0025] Furthermore, if data migration is complete but data reconstruction is not complete, do the following:

[0026] According to the repair ratio threshold, the data blocks to be repaired that have not been subjected to the data reconstruction operation are divided again, and data migration and data reconstruction are performed on the divided data blocks to be repaired respectively, and vice versa.

[0027] Furthermore, when a new object is written into the system, the data blocks and the check blocks are grouped into coding stripes, and the coding stripes are stored on multiple hard disks respectively, and each coding stripe has a unique ID;

[0028] When requesting to read data, locate the data block according to the ID and offset of the encoding stripe, and read the data block;

[0029] When updating data, the data block to be updated is read through the ID and offset of the encoded stripe, and the new data is written into the data block and synchronized to the cloud.

[0030] A second aspect of the present invention provides a cloud-edge collaborative data fault-tolerant system, which applies a cloud-edge collaborative data fault-tolerant method described in the first aspect, including a cloud server, wherein the cloud server includes:

[0031] Check block storage node: responsible for storing check blocks, using log-style writing to balance memory overhead and update efficiency;

[0032] Data storage node: periodically receives data sent by IoT terminals and stores data using erasure code storage;

[0033] Metadata storage and computing nodes: manage metadata information of all data, including object index, stripe index, and detailed metadata of data blocks; and are responsible for collaborating with fault prediction computing nodes to calculate thresholds for data migration and repair;

[0034] Fault prediction computing node: Regularly receives hard disk SMART information uploaded by IoT devices and predicts hard disk failures. When a hard disk is judged to be about to fail, data migration and reconstruction operations are initiated.

[0035] Furthermore, the cloud-edge collaborative data fault-tolerant system further includes an Internet of Things terminal:

[0036] The data blocks of the plurality of IoT terminals form a stripe of the same erasure code. When the data block reaches a coding quantity threshold, the data block is calculated using the erasure code to obtain redundant data of the data block as a check block.

[0037] The data block and the check block form a new coding stripe. Each coding stripe has a corresponding number of coding blocks, and each coding block has a unique stripe ID.

[0038] The check block, stripe ID, and the offset of the data of multiple IoT terminals within the stripe are all sent asynchronously to the cloud server. Compared with the prior art, the cloud-edge collaborative data fault-tolerant method and system described in the present invention has the following beneficial effects:

[0039] Through the cloud-edge collaborative data fault-tolerance method, technicians can be notified in advance to take measures and proactively repair the data in these hard drives before the failure actually occurs, thereby ensuring data security and reducing maintenance costs.

[0040] Regularly monitor the health status of the hard disk, perform fault repair and maintenance in a timely manner, extend the service life of the hard disk, improve the stability and performance of the entire system, and provide real-time monitoring and reporting functions. Users can understand the health status and fault prediction of the hard disk at any time, so as to take timely response measures.

[0041] The IoT terminal continuously collects business data and stores it locally, calculates the redundant data (i.e., check blocks) corresponding to the business data through erasure codes, and stores it in the cloud. Based on this architecture, by accurately predicting hard drive failures, conditions are created for active data repair, thereby ensuring the reliability of IoT data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of the execution flow of a cloud-edge collaborative data fault-tolerant method according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the cloud-edge collaborative data fault-tolerant system architecture according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a data reading and writing operation flow according to an embodiment of the present invention;

[0046] Figure 4 The figure is a schematic diagram of a data recovery operation flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0048] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0050] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0051] like Figure 1-4 As shown: A cloud-edge collaborative data fault-tolerant method, comprising:

[0052] S1. Build and train two fault prediction models with different accuracies. The two fault prediction models are located in the IoT terminal and the cloud respectively.

[0053] S2, collect hard disk SMART data and add time series features and trend features to it, and input the SMART data into the fault prediction model located in the IoT terminal to perform preliminary fault prediction;

[0054] S3. When the preliminary prediction result is that the hard disk is on the verge of failure, the SMART data with time series characteristics and trend characteristics is sent to the failure prediction model located in the cloud for secondary failure prediction;

[0055] S4. When the secondary prediction result indicates that the hard disk is on the verge of failure, the hard disk is repaired before the hard disk fails by performing data migration and reconstruction operations in parallel.

[0056] The role of the cloud-edge collaborative data fault-tolerant method is: on the one hand, two fault prediction models of different sizes and precisions are trained, and the small and low-precision model is distributed on each consumer electronic product. Only when the small model warns that the hard disk may fail, the SMART data of the hard disk is transmitted to the cloud, and the high-precision large model is used again for the final prediction; on the other hand, a data repair method based on the short board theory of multiple disks is used for joint active repair, that is, combining migration and reconstruction to jointly repair healthy disks, faulty disks, and cloud check blocks. The IoT terminal continuously collects business data and stores it locally, calculates redundant data (i.e., check blocks) corresponding to the business data by means of erasure codes, and stores them in the cloud;

[0057] When the initial prediction result is that the hard disk is on the verge of failure, the hard disk SMART data is sent to the cloud. If the result of the secondary failure prediction performed on the cloud is still that the hard disk is on the verge of failure, the locally stored business data and the redundant data in the cloud are used to perform data migration and reconstruction operations.

[0058] The process of training two fault prediction models with different precisions in step S1 includes:

[0059] The hard disk failure model is trained by combining the gradient boosting decision tree model with the training set, and the model is tuned through grid search and cross-validation to obtain a lightweight fault prediction model and a full-scale fault prediction model.

[0060] The lightweight fault prediction model is distributed on each IoT terminal, and the full-scale fault prediction model is uploaded to the cloud.

[0061] Data stability is maintained through time series features and trend features. The time series features are used to evaluate the time series complexity of hard disk SMART data, and the trend features are used to reflect the change trend of hard disk SMART data; wherein the time series features are introduced through a sliding window method.

[0062] Considering the time series of hard disk data, corresponding time series statistical features CID (Complexity Invariant Distance) and MA (Moving Average) are constructed for each SMART data. CID is used to evaluate the complexity of the time series, and MA is used to reflect the changing trend of the hard disk SMART data. After the introduction of time series features, the accuracy of the model is improved, and the model can predict hard disk failures earlier.

[0063] The process of performing data migration and reconstruction operations includes:

[0064] Select healthy hard disks with free capacity that meets the needs of migration and reconstruction, set a threshold for the repair ratio, and divide the data blocks to be repaired into two parts: migration and reconstruction according to the threshold;

[0065] By adjusting the repair ratio threshold, the execution time of data migration is close to that of data reconstruction.

[0066] Data migration is performed on the data blocks to be repaired in the migration part, and data reconstruction is performed on the data blocks to be repaired in the reconstruction part.

[0067] The data migration adopts a direct copy method to perform data repair, copying the data of the hard disk on the verge of failure to the target healthy disk;

[0068] The data reconstruction uses an erasure code method to repair the data, and uses the reconstructed hard disk group to perform encoding and decoding calculations to obtain corresponding data block content and store it in a target healthy disk.

[0069] If data migration is complete but data reconstruction is not complete, do the following:

[0070] According to the repair ratio threshold, the data blocks to be repaired that have not been subjected to the data reconstruction operation are divided again, and data migration and data reconstruction are performed on the divided data blocks to be repaired respectively, and vice versa.

[0071] In some embodiments, the specific execution process of the above steps S1-S4 is as follows:

[0072] Data preprocessing:

[0073] Capture data and relabel: Use the backblaze dataset as the initial dataset. First, capture the data in a time window of N for the hard disk in the initial dataset. For a faulty hard disk, take the data of the last N days and mark these data as 1, which means that a fault will occur in the time window. For a healthy disk, first remove the data in the last time window, then randomly select the data in a time window in the previous time period and mark these data as 0.

[0074] Feature screening: Features with all null values ​​and variance of 0 in the data set are removed.

[0075] Fill missing values: Use the mode of the 10 days before and after the missing value to fill.

[0076] Normalization: Use min-max normalization to limit the value range to [0,1], specifically: ;

[0077] Here, x is the original value, x_min and x_max are the minimum and maximum values ​​of the attribute in the dataset, respectively.

[0078] Add time series features: For each SMART attribute CID feature and MA feature after preprocessing, the time window of CID and MA features is 7 days.

[0079] Model training and tuning: GBDT is used in combination with the preprocessed training set to train the hard disk fault prediction model. GridSearch is used in combination with the CrossValidation method to find the optimal hyperparameters to tune the model. Two models are trained. One is lightweight, occupies less space, but has lower accuracy, and is a lightweight fault prediction model. The other is full-scale, occupies more space, has higher accuracy, and is a full-scale fault prediction model. When training the lightweight fault prediction model, we randomly delete the negative samples of the initial data set so that the final positive and negative sample ratio becomes 1:1, which makes the data set size become 1 / 22 of the original.

[0080] Hard disk failure prediction:

[0081] Input the collected hard disk SMART data into the fault prediction model to first obtain the sample-level prediction result; map the sample-level prediction result to the disk-level prediction result; and use map_function to act on the sample-level result to obtain the disk-level prediction result, that is, whether the hard disk will fail.

[0082] Based on the prediction of the time window and mapping function, the time window size for the hard disk data is N, and the data of a certain hard disk is D={1,2,…,}. Assuming that the trained model is g(x), the hard disk data D to be predicted is input into the trained model to obtain the time series {yprob1,yprob2,…,yprobN} of the probability of failure of this hard disk;

[0083] Among them, yprobi,i∈[1,N] is a continuous variable with a value range between [0,1], which can be interpreted as the probability of failure occurring within the time window;

[0084] Similarly, the threshold α is used to determine whether the hard disk will fail, and {ypredict1, ypredict2, …, ypredictN} is obtained;

[0085] Finally we get:

[0086] Ypredict=map_function(ypredict1,ypredict2,…,ypredictN), that is, whether this hard disk will fail in the future time window.

[0087] If the lightweight fault prediction model determines that there is a suspected fault, the SMART data is sent to the cloud, and the full-scale fault model determines whether there is an imminent fault. If it is still determined to be an imminent fault, the process proceeds to step S43.

[0088] Data recovery operation:

[0089] After hard disk D is predicted to be on the verge of failure, active fault-tolerant data repair is immediately initiated;

[0090] Determine the same group of healthy hard disks and cloud check blocks that participate in the joint repair, which is called reconstruction group A;

[0091] Select a healthy hard disk with sufficient free capacity as the repair target disk;

[0092] Initialize the repair ratio threshold y = 1 / 2. Assume that the hard disk D on the verge of failure has n data blocks to be repaired, data migration is responsible for repairing n×y data blocks, and data reconstruction is responsible for repairing n×(1-y) data blocks.

[0093] The repair of data migration directly copies the data block to the target disk, and the repair of data reconstruction performs encoding and decoding calculations on the reconstruction group A to obtain the corresponding data block content and stores it in the target disk.

[0094] If data migration is completed but the rebuilt data blocks have not been repaired, 1 / 2 of the remaining data blocks will be completed by data migration, and so on, and vice versa.

[0095] If the hard disk fails before data migration is completed, reconstruction team A will be responsible for all remaining data repair tasks.

[0096] The threshold y is adjusted appropriately to make the repair time of migrated data exactly equal to the repair time of reconstructed data.

[0097] When a new object is written into the system, the data blocks and check blocks are grouped into coding stripes, and the coding stripes are stored on multiple hard disks respectively. Each coding stripe has a unique ID.

[0098] In some embodiments, the specific execution process when a new object is written to the system is a write operation method, including: when a new object is written to the system, the metadata server maintains a queue, which encodes the collected objects; in the system, the size of each data block is 4KB, and once the encoding conditions are met, the metadata server generates a check block, and these data blocks and the check block constitute a new encoded stripe; each stripe has a unique stripe ID, which is stored in the index of the key-value storage; the object is located by the stripe ID in the index and the offset within the stripe; the data blocks are sent to k mechanical hard disks respectively, and metadata information such as m check blocks, stripe ID, and the offset of each terminal data in the stripe are sent to the cloud server in an asynchronous manner.

[0099] When a data read request is made, the data block is located according to the ID and offset of the encoding stripe, and the data block is read.

[0100] In some embodiments, the specific execution process of the request to read data is a read operation method, including: when a user requests to read data, the user first sends a query request to the metadata server, then retrieves the stripe ID and offset from the key-value storage index of the metadata server, and finally reads the data block from the corresponding IoT terminal hard disk.

[0101] When updating data, the data block to be updated is read through the ID and offset of the encoded stripe, and the new data is written into the data block and synchronized to the cloud.

[0102] In some embodiments, the specific execution process of updating data is an update operation method, including: when updating an object, the metadata server first obtains the corresponding stripe ID, offset and length from the storage index; then the metadata server reads the old data block and check block from the stripe; the local hard disk node storing the data block determines whether the data block has been updated. If it is the first update, read the old data block, write the new data to the local hard disk and attach the version number, and append the old data and the new data together to the log space reserved on the cloud node; if the data block is not updated for the first time, the metadata server directly writes the data block with the new version number attached, and then appends the new data to the log space reserved for the cloud node.

[0103] A cloud-edge collaborative data fault-tolerant system, applying the above-mentioned cloud-edge collaborative data fault-tolerant method, includes a cloud server, and the cloud server includes:

[0104] Check block storage node: responsible for storing check blocks, using a log-style writing method to balance memory overhead and update efficiency; using high-speed persistent storage media such as SSD or NVM to store check data to improve data read and write speed and system response performance.

[0105] Data storage node: periodically receives data sent by IoT terminals to prevent the terminal storage capacity from being exhausted; under the erasure code storage mode, only the full amount of data from some terminals needs to be sent to meet data backup and recovery needs, reducing the network bandwidth and storage space occupied.

[0106] Metadata storage and computing node: manages metadata information of all data, including object index, stripe index and detailed metadata of data blocks (such as offset, length, etc.); at the same time, this node is also responsible for collaborating with the fault prediction computing node to calculate the ratio of data migration and repair and optimize the system's fault tolerance strategy.

[0107] Fault prediction computing node: Regularly receives hard disk SMART information uploaded by IoT devices. Each IoT terminal first performs a preliminary fault prediction. If the hard disk is suspected to be faulty, the pre-processed SMART information is sent to the fault prediction computing node. The cloud model accurately analyzes the data. If it predicts that the hard disk is about to fail, the system will actively perform data repair operations before the failure occurs.

[0108] The cloud-edge collaborative data fault-tolerant system further includes an Internet of Things terminal, which includes a sensor device, a computing unit, a network transmission device (wired or wireless) and a mechanical hard disk;

[0109] The data of k IoT terminals constitute the same erasure coded stripe. As the terminal data is continuously generated, once the data reaches the coding quantity threshold, the data of the k terminals are jointly calculated to generate m check blocks; these data blocks and check blocks constitute a new coding stripe, each of which has (k+m) identical coding blocks and a unique stripe ID; the check blocks, together with the stripe ID, the offset of each terminal data in the stripe and other metadata information are sent to the cloud server in an asynchronous manner.

[0110] To facilitate understanding of the technical solutions described in the above specific embodiments, a cloud-edge collaborative data fault-tolerant method and system is provided, with specific execution process and corresponding data:

[0111] (1) Data writing: The system collects data at the terminal and divides it into 4 data blocks, each of which is 4KB in size. The metadata server calculates and generates 2 check blocks. The data block and check block form a coded stripe and generate a unique stripe ID: SDRT1233. The check block, stripe ID: SDRT1233 and the offset of the terminal data in the stripe are sent to the cloud server asynchronously.

[0112] (2) User read: The user sends a query request to the metadata server, and the metadata server returns the retrieval stripe ID: SDRT1233 and offset, and finally reads the data block from the IoT terminal hard disk and returns it.

[0113] (3) Data update: The metadata server first obtains the stripe ID of the data block to be updated: SDRT1233, offset and length from the storage index; then the metadata server reads the old data block and check block from the stripe, detects that the data block is updated for the first time, reads the old data block, writes the new data to the local hard disk and appends the version number, and finally appends the old data and the new data to the log space reserved on the cloud node; when updating later, the data block with the new version number is written directly to the metadata server, and then the new data is appended to the space reserved on the cloud node.

[0114] (4) Fault warning

[0115] a) First, in the Backblaze 2019 dataset, two different models are selected from two manufacturers, namely ST12000NM0007, ST4000DM000, HUH721212ALN604, and HMS5C4040ALE640. The specific situation is shown in the following table:

[0116]

[0117] b) Data preprocessing: For the hard disk in the initial data set, data within a time window is intercepted, and the time window size N is 10 days. For the faulty hard disk, the data of the last 10 days is taken and marked as 1; for the healthy disk, the data in the last time window is first removed, and then the data in a time window in the previous time period is randomly selected and marked as 0. The features with all nulls and variances of 0 are removed, and the remaining attributes are shown in the following table:

[0118]

[0119] The missing values ​​in the data are filled by the mode of the 10 days before and after the missing value. Min-max normalization is used to limit the value range to [0,1]. CID features and MA features are added to each SMART attribute after preprocessing, and the time window is 7 days. The final data set has a total of 121 features, including 41 SMART attributes, 39 CID features and 41 MA features.

[0120] In order to capture the mutation before the hard disk fails, CID (complexity invariant distance) is introduced, as shown in the following table:

[0121]

[0122] By comparing the CIDs of randomly selected healthy disks and faulty disks, it is found that the CID of the faulty disk is significantly higher than that of the healthy disk in terms of the SMART attributes that have been mutated.

[0123] Using CID to measure the time series complexity of hard disk SMART data can accurately distinguish healthy disks from faulty disks in certain attributes. However, one of the shortcomings of the two-stage hard disk failure prediction method is that the model only judges the status of the hard disk on that day through the SMART data of the hard disk on that day. The failure of a hard disk is a gradual process. If only the SMART data of the hard disk on that day is considered, the accuracy of the model will be low.

[0124] In order to solve this problem, the sliding window method is used in the data set to add the time series feature CID to measure the complexity of the time series. For example, cid_smart_1_raw means that the sliding window is applied to the SMART attribute smart_1_raw. At the same time, this article also adds MA (Moving Average) to reflect the trend of data changes, as shown in the following formula; .

[0125] c) Use GBDT combined with the preprocessed training set to train the hard disk failure prediction model: Use GridSearch combined with the CrossValidation method to find the optimal hyperparameters of the GBDT algorithm, so as to tune the model. The K-foldCrossValidation method is used to evaluate the prediction effect of the model, where the parameter K is set to 5, that is, the data set is divided into K parts, K-1 of which are used as training sets, and the remaining 1 is used as a test set. Finally, the average mean square error is taken to evaluate the model. GridSearch searches for the optimal hyperparameters of the model through exhaustive enumeration, so as to find the parameters with the highest accuracy of the model on the test set among all the parameters. The following table lists the hyperparameters, value ranges, and optimal hyperparameters of the LGB model after grid search:

[0126]

[0127] Among them, adjusting num_leaves, max_depth, reg_alpha, and reg_lambda is used to deal with the problem of model overfitting; adjusting boosting_type, learning_rate, and n_estimators is to improve the accuracy of the model.

[0128] d) After the above steps, two models in plk format are output, namely the lightweight fault prediction model and the full fault prediction model. The training data set used by the lightweight fault prediction model is the negative samples of the initial data set, which are randomly deleted so that the final positive and negative sample ratio becomes a lightweight data set with a 1:1 ratio. Then, the lightweight fault prediction model is distributed to each IoT terminal, and the full fault prediction model is saved in the cloud.

[0129] e) Prediction based on time window and mapping function

[0130] Input the SMART data of the hard disk of IoT terminal A into the lightweight fault prediction model, and obtain the probability time series {0,0,0,0,0} of hard disk failure. Finally, we get Y predict =map_function(0,0,0,0,0)=0, which means that this hard disk is unlikely to fail in the short term.

[0131] Input the SMART data of the hard disk of IoT terminal B into the lightweight fault prediction model, and obtain the probability time series of hard disk failure {0.3, 0.5, 0.7, 0.9, 0}, and finally obtain Y predict=map_function(0.3,0.5,0.7,0.9,0)=1, which means that the hard disk is suspected to be about to fail. Then, all the collected SMART data are preprocessed and simplified, and then input into the full-scale failure prediction model to obtain the probability time series of hard disk failure {0,0,0,0,0.7}, and finally obtain Y predict =map_function(0,0,0,0,0)=0, this hard disk will not fail in the short term

[0132] Input the SMART data of the hard disk of IoT terminal C into the lightweight fault prediction model, and obtain the probability time series of hard disk failure {0.3, 0.5, 0.7, 0.9, 0}, and finally obtain Y predict =map_function(0.3,0.5,0.7,0.9,0)=1, which means that the hard disk is suspected to be about to fail. Then, all the collected SMART data are preprocessed and simplified, and then input into the full-scale failure prediction model to obtain the probability time series of hard disk failure {0,0,0,0,0.7}, and finally obtain Y predict =map_function(0,0,0,0,0.7)=1, that is, the hard disk is about to fail, and the data recovery operation begins

[0133] (5) Data recovery

[0134] Hard disk C is predicted to fail soon, and the repair target disk is M. Select disk A for joint repair and form a reconstruction group with the cloud check block. Take the repair ratio threshold y=1 / 2, data migration is responsible for the repair of 1 / 2 of the data blocks, and the data blocks are directly copied to the target disk M; data reconstruction is responsible for the repair of the remaining data blocks, and the corresponding data block content is obtained through the encoding and decoding calculation of the reconstruction group and stored in the target disk M. When the data migration is completed, the reconstructed data blocks have not been repaired, and the remaining 1 / 2 of the data blocks are completed by data migration.

[0135] Another hard disk D is predicted to fail soon, and the repair target disk is M. Disk B is selected for repair together, and forms a reconstruction group with the cloud check block. The repair ratio threshold y=1 / 2 is taken for repair, but hard disk D fails before the data migration is completed. The remaining data blocks are all repaired by the reconstruction group and stored in the target disk M.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-edge collaborative data fault-tolerance method, characterized in that: include: Build and train two fault prediction models with different accuracies, one in the IoT terminal and the other in the cloud. Collect hard disk SMART data and add time series features and trend features to it, and input the SMART data into the fault prediction model located in the IoT terminal for preliminary fault prediction; When the initial prediction result shows that the hard disk is on the verge of failure, the SMART data with time series characteristics and trend characteristics will be sent to the failure prediction model in the cloud for secondary failure prediction; When the secondary prediction result is that the hard disk is on the verge of failure, the hard disk is repaired before the hard disk fails by performing data migration and reconstruction operations in parallel.

2. A cloud-edge collaborative data fault-tolerant method according to claim 1, characterized in that: The IoT terminal continuously collects business data and stores it locally, calculates redundant data corresponding to the business data by erasure coding, and stores it in the cloud; When the initial prediction result is that the hard disk is on the verge of failure, the hard disk SMART data is sent to the cloud. If the result of the secondary failure prediction performed on the cloud is still that the hard disk is on the verge of failure, the locally stored business data and the redundant data in the cloud are used to perform data migration and reconstruction operations.

3. According to the cloud-edge collaborative data fault-tolerant method of claim 1, it is characterized in that: The process of training two fault prediction models with different precisions includes: The hard disk failure model is trained by combining the gradient boosting decision tree model with the training set, and the model is tuned through grid search and cross-validation to obtain a lightweight fault prediction model and a full-scale fault prediction model. The lightweight fault prediction model is distributed on each IoT terminal, and the full-scale fault prediction model is uploaded to the cloud.

4. A cloud-edge collaborative data fault-tolerant method according to claim 3, characterized in that: Data stability is maintained through time series features and trend features. The time series features are used to evaluate the time series complexity of hard disk SMART data, and the trend features are used to reflect the change trend of hard disk SMART data; wherein the time series features are introduced through a sliding window method.

5. The cloud-edge collaborative data fault-tolerant method according to claim 1, characterized in that: The process of performing data migration and reconstruction operations includes: Select healthy hard disks with free capacity that meets the needs of migration and reconstruction, set a threshold for the repair ratio, and divide the data blocks to be repaired into two parts: migration and reconstruction according to the threshold; By adjusting the repair ratio threshold, the execution time of data migration is close to that of data reconstruction. Data migration is performed on the data blocks to be repaired in the migration part, and data reconstruction is performed on the data blocks to be repaired in the reconstruction part.

6. A cloud-edge collaborative data fault-tolerant method according to claim 5, characterized in that: The data migration adopts a direct copy method to perform data repair, copying the data of the hard disk on the verge of failure to the target healthy disk; The data reconstruction uses an erasure code method to repair the data, and uses the reconstructed hard disk group to perform encoding and decoding calculations to obtain corresponding data block content and store it in a target healthy disk.

7. A cloud-edge collaborative data fault-tolerant method according to claim 6, characterized in that: If data migration is complete but data reconstruction is not complete, do the following: According to the repair ratio threshold, the data blocks to be repaired that have not been subjected to the data reconstruction operation are divided again, and data migration and data reconstruction are performed on the divided data blocks to be repaired respectively, and vice versa.

8. The cloud-edge collaborative data fault-tolerant method according to claim 2, characterized in that: When a new object is written into the system, the redundant data is used as a check block, and the data block and the check block are combined into a coding stripe, which is then stored on multiple hard disks. Each coding stripe has a unique ID. When requesting to read data, locate the data block according to the ID and offset of the encoding stripe, and read the data block; When updating data, the data block to be updated is read through the ID and offset of the encoded stripe, and the new data is written into the data block and synchronized to the cloud.

9. A cloud-edge collaborative data fault-tolerant system, applying a cloud-edge collaborative data fault-tolerant method according to any one of claims 1 to 8, characterized in that: The cloud server includes: Check block storage node: responsible for storing check blocks, using log-style writing to balance memory overhead and update efficiency; Data storage node: periodically receives data sent by IoT terminals and stores data using erasure code storage; Metadata storage and computing nodes: manage metadata information of all data, including object index, stripe index, and detailed metadata of data blocks; and are responsible for collaborating with fault prediction computing nodes to calculate thresholds for data migration and repair; Fault prediction computing node: Regularly receives hard disk SMART information uploaded by IoT devices and predicts hard disk failures. When a hard disk is judged to be about to fail, data migration and reconstruction operations are initiated.

10. A cloud-edge collaborative data fault-tolerant system according to claim 9, characterized in that: Also includes IoT terminals: The data blocks of the plurality of IoT terminals form a stripe of the same erasure code. When the data block reaches a coding quantity threshold, the data block is calculated using the erasure code to obtain redundant data of the data block as a check block. The data block and the check block form a new coding stripe. Each coding stripe has a corresponding number of coding blocks, and each coding block has a unique stripe ID. The check block, the stripe ID, and the offset of the data of multiple IoT terminals within the stripe are all sent asynchronously to the cloud server.

Images