Fault data recovery method and device, computer equipment and storage medium

Abstract

The invention relates to a fault data recovery method and device, computer equipment and a storage medium, and belongs to the technical field of distributed storage systems.The method comprises the steps that obtained original data is cut into a plurality of data blocks, the data blocks are grouped, and a plurality of stripe groups are obtained; performing erasure code coding on the data blocks in the strip groups to generate corresponding check blocks; determining an abnormal data block according to the health state of the data block; abnormal data blocks among the strip groups are adjusted; updating the adjusted verification block corresponding to each band group; and in response to the fault of the abnormal data block in the stripe group, recovering the data of the fault data block according to the updated verification block. According to the method, the data recovery time can be remarkably shortened, so that the high-efficiency operation and the data security of the system are ensured while the data reliability and the fault tolerance are improved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed storage systems, and in particular to a method, apparatus, computer equipment and storage medium for recovering fault data. Background Art

[0002] With the development of technologies such as big data, cloud computing and the Internet of Things, the popularization of smart devices, and the rapid growth of data volume and data types, higher demands are placed on large-scale and diversified data storage systems. Taking distributed storage systems as an example, distributed storage systems achieve high-performance, high-reliability and scalable data storage by storing data on multiple independent nodes and performing distributed management. However, when data fails and needs to be recovered, it is necessary to complete the query and recovery of the faulty data in massive data. Currently, distributed storage systems mainly have two important data redundancy protection mechanisms: erasure codes and multiple copies. Both mechanisms require a long data recovery time to solve the data recovery problem, especially when using erasure codes. When data or check blocks are lost, it is necessary to download data from multiple storage nodes and perform encoding and decoding operations to recover the lost data blocks. This process involves a large amount of data reading and calculation, which leads to long recovery time and low efficiency for faulty data. Summary of the invention

[0003] Based on this, it is necessary to provide a fault data recovery method, device, computer equipment and storage medium that can shorten the recovery time of fault data and improve efficiency in response to the above technical problems.

[0004] In a first aspect, a method for recovering fault data is provided, the method comprising:

[0005] Get the original data;

[0006] Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0007] Performing erasure coding on the data blocks in the stripe grouping to generate check blocks that match the data blocks in each stripe grouping;

[0008] Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0009] Adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0010] Update and adjust the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0011] In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0012] Optionally, cutting the original data into multiple data blocks includes:

[0013] Get the data length and initial block length value of the original data;

[0014] Determine whether the length of the original data is an integer multiple of the initial block length value;

[0015] In response to the length of the original data not being an integer multiple of the initial block length value, adding a placeholder at the end of the original data until the length of the original data is an integer multiple of the initial block length value;

[0016] In response to the length of the original data being an integer multiple of the initial block length value, an erasure coding algorithm is used to cut the original data into a plurality of data blocks according to the initial block length value.

[0017] Optionally, the plurality of data blocks are grouped to obtain a plurality of stripe groups including;

[0018] Multiple data blocks are grouped according to a target grouping value to obtain multiple stripe groups, wherein each stripe group contains the same number of data blocks with equal data size and the same number of check blocks matching the data blocks, and the target grouping value is used to describe the number of data blocks contained in each stripe group.

[0019] Optionally, the method further includes storing stripe grouping and check blocks:

[0020] The data blocks in the plurality of stripe groups are stored in different object storage devices respectively;

[0021] storing the check blocks that match the data blocks in each stripe group in an object storage device that is different from the storage location of the data blocks in the plurality of stripe groups;

[0022] Among them, object storage devices are stored in different system storage nodes.

[0023] Optionally, obtaining the health status of data blocks in each stripe group includes:

[0024] Obtain historical data of the object storage device and preprocess the historical data, which is used to describe the specifications and performance of the object storage device;

[0025] Extract the characteristic values ​​of the preprocessed historical data;

[0026] Divide the historical data feature values ​​into training sets and test sets according to the preset ratio;

[0027] Select the initial logistic regression model;

[0028] The initial logistic regression model is trained using the training set to obtain a first logistic regression model;

[0029] Adjusting the parameter values ​​of the first logistic regression model to obtain a second logistic regression model;

[0030] The second logistic regression model is verified and evaluated using the test set. According to the evaluation results, the target logistic regression model is determined. The expression of the target logistic regression model includes:

[0031]

[0032] Among them, m represents the time period, x i represents the probability of product failure in the i-th hour, f(x) represents the failure rate of the object storage device process, and p(x) represents the output value of the logistic regression model;

[0033] The health status of the data blocks in each stripe group is determined based on the output value of the target logistic regression model.

[0034] Optionally, determining the health status of data blocks in each stripe group according to the output value of the target logistic regression model includes:

[0035] Compare the output value of the target logistic regression model with the preset threshold;

[0036] In response to an output value of the target logistic regression model being greater than a preset threshold, a data block corresponding to the output value of the target logistic regression model is defined as an abnormal data block.

[0037] Optionally, adjusting the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe group containing the abnormal data blocks does not exceed the number of matched check blocks includes:

[0038] Execute the abnormal data block number adjustment process:

[0039] Acquire multiple stripe groups containing abnormal data blocks, and select a second target stripe group according to the number of abnormal data blocks in the multiple stripe groups;

[0040] Compare the number of abnormal data blocks in the second target stripe group with the number of check blocks that match the data blocks in the second target stripe group:

[0041] In response to the number of abnormal data blocks in the second target stripe group being less than the number of check blocks matching the data blocks in the second target stripe group, exchanging any abnormal data block in the third target stripe group with a normal data block in the second target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks matching the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the third target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group;

[0042] In response to the number of abnormal data blocks in the second target stripe group being greater than the number of check blocks that match the data blocks in the second target stripe group, exchanging any abnormal data block in the second target stripe group with a normal data block in a fourth target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks that match the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the fourth target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group, and the number of abnormal data blocks in the fourth target stripe group is less than the number of check blocks that match the data blocks in the fourth target stripe group;

[0043] Repeat the process of adjusting the number of abnormal data blocks until the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group whose number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

[0044] In a second aspect, a fault data recovery device is provided, the device comprising:

[0045] A first acquisition module is used to acquire original data;

[0046] A grouping module is used to cut the original data into multiple data blocks, and group the multiple data blocks to obtain multiple stripe groups;

[0047] The encoding module is used to perform erasure coding on the data blocks in the stripe grouping to generate a check block matching the data blocks in each stripe grouping;

[0048] The second acquisition module is used to acquire the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0049] An adjustment module is used to adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0050] An updating module, used for updating the check blocks matched by the data blocks in each stripe group after adjusting the abnormal data blocks;

[0051] The data recovery module is used for, in response to a failure of an abnormal data block in the first target stripe group, recovering data of the failed data block according to a check block corresponding to the updated first target stripe group.

[0052] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0053] Get the original data;

[0054] Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0055] Performing erasure coding on the data blocks in the stripe grouping to generate check blocks that match the data blocks in each stripe grouping;

[0056] Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0057] Adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0058] Update and adjust the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0059] In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0060] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0061] Get the original data;

[0062] Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0063] Performing erasure coding on the data blocks in the stripe grouping to generate check blocks that match the data blocks in each stripe grouping;

[0064] Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0065] Adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0066] Update and adjust the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0067] In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0068] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0069] Get the original data;

[0070] Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0071] Performing erasure coding on the data blocks in the stripe grouping to generate check blocks that match the data blocks in each stripe grouping;

[0072] Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0073] Adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0074] Update and adjust the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0075] In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0076] The above-mentioned fault data recovery method, device, computer equipment and storage medium, the method includes: obtaining original data; cutting the original data into multiple data blocks, grouping the multiple data blocks to obtain multiple stripe groups; encoding the data blocks in the stripe groups with correction and erasure codes to generate check blocks that match the data blocks in each stripe group; obtaining the health status of the data blocks in each stripe group, and determining the abnormal data blocks according to the health status of the data blocks; adjusting the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks; updating the check blocks that match the data blocks in each stripe group after adjusting the abnormal data blocks; in response to the abnormal data block in the first target stripe group failing, correcting the data of the failed data block according to the check blocks corresponding to the updated first target stripe group Recovery: The present application achieves more efficient data management and storage by cutting the original data into multiple data blocks and performing striping grouping on the data blocks, and encodes the data blocks in each stripe grouping to generate check blocks. The increase of such redundant information provides a guarantee for subsequent data recovery; the health status of the data blocks in the stripe grouping is monitored to facilitate timely discovery of abnormal data blocks. After the abnormality is discovered, the distribution of the abnormal data blocks is adjusted to ensure that the number of abnormal data blocks in each stripe grouping does not exceed the number of matched check blocks, which can effectively prevent excessive abnormal data in a single stripe and ensure the overall recoverability of the data. This adjustment mechanism increases the system's tolerance for faults, and restores faulty data based on updated check blocks. On the one hand, it can quickly locate the faulty data, and on the other hand, it can quickly restore lost or damaged data blocks with the help of check blocks. The overall technical solution can significantly shorten the data recovery time, thereby improving data reliability and fault tolerance while ensuring efficient system operation and data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 The overall architecture diagram of a distributed storage system of a method for recovering fault data in one embodiment is shown;

[0078] Figure 2 A schematic diagram of a process flow of a method for recovering fault data in an embodiment;

[0079] Figure 3 Another flowchart of a method for recovering fault data in one embodiment;

[0080] Figure 4 is a structural block diagram of a fault data recovery device in one embodiment;

[0081] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0084] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0085] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0086] The rise of big data and cloud computing technologies has provided more application scenarios for distributed storage systems and promoted the development of distributed storage system technology. A distributed storage system refers to a system that stores data on multiple independent nodes and manages it in a distributed manner, aiming to provide high-performance, high-reliability and scalable storage solutions. The design concept of a distributed storage system is based on scalability and fault tolerance. It adopts a distributed architecture, slices data and stores it in the form of objects on multiple nodes, and ensures the capacity and load balance of each node. It uses a technology for data access and management over the network, and provides a unified storage interface. Among them, high reliability: using erasure code or replica technology for data redundancy and automatic fault recovery mechanism can ensure data reliability and persistence; high scalability: distributed storage system clusters can be dynamically expanded horizontally to support hundreds or even thousands of nodes; unified interface: providing a unified file system, object storage and block storage interface to facilitate user data access and management.

[0087] The development trend of distributed storage system technology includes more intelligent data management and scheduling, more efficient data redundancy, more secure and reliable data protection and backup technology, etc. Erasure Coding and Replica are two important data redundancy protection mechanisms. Erasure Coding protects data by dividing data into multiple fragments and generating additional redundant fragments (i.e., encoding). Therefore, even if some original fragments (data blocks) are lost, the complete data can still be restored from the remaining fragments. The replica mechanism achieves data redundancy by storing the same data copies on multiple nodes. When a node fails or data is lost, the data can be read directly from the replicas on other nodes.

[0088] According to the background technology, with the development of technologies such as big data, cloud computing and the Internet of Things, data protection and fault recovery technology of distributed storage systems will face new challenges and opportunities. Erasure code technology achieves data redundancy and fault tolerance by dividing data into blocks and calculating check blocks. Although this technology has lower storage overhead than traditional data copy technology, it also brings a series of challenges, one of which is the long data recovery time. Specifically, when data or check blocks are lost, it is necessary to download data from multiple storage nodes and perform encoding and decoding operations to restore the lost data blocks. This process involves a large amount of data reading and calculation, which leads to slow data recovery. In addition, the data repair overhead of erasure code technology is large. When a coding block fails, multiple other coding blocks need to be used for repair, which further increases the complexity and time cost of data recovery. Therefore, although erasure code technology has advantages in reducing storage costs, its disadvantage of long data recovery time needs to be fully considered and solved in practical applications.

[0089] Methods for improving the failure recovery speed of the erasure storage pool of a distributed storage system include adding redundant data. By adding redundant data, the failure recovery speed can be improved. When a storage node fails, the system can quickly recover through redundant data; or by using a parallel recovery method, multiple nodes can be used for data recovery at the same time, thereby improving the recovery speed. For example, when a disk fails, the data recovery operation is divided into several small parts for operation, which includes a two-stage process, namely, the initial layout stage of data blocks and the recovery stage of failed blocks. In the first stage, when the data is initially stored in the storage system, all data blocks are distributed to each node. In the second stage, in order to perform efficient batch recovery, parallel recovery is performed simultaneously, and then different recovery schemes are implemented according to the characteristics of different data blocks, with the purpose of dispersing and balancing the IO (Input / Output) traffic generated by data recovery across racks; or by using an intelligent data distribution algorithm, data is evenly distributed on different storage nodes, thereby improving the efficiency of failure recovery; but the above schemes have the following problems:

[0090] (1) Increased costs: Adding redundant data and adopting parallel recovery methods will increase system costs, including storage costs and computing costs;

[0091] (2) Data consistency: Parallel recovery may cause data consistency issues, and additional mechanisms are needed to ensure data consistency;

[0092] (3) Network bandwidth limitation: When performing data recovery, the network bandwidth may be limited, thus affecting the recovery speed.

[0093] To solve the above technical problems, the present application provides a fault data recovery method, apparatus, device and storage medium, which cuts the original data into multiple data blocks, groups the data blocks into stripes, monitors the health status of the data blocks in the stripe groups, and after an abnormality is found, adjusts the distribution of the abnormal data blocks to ensure that the number of abnormal data blocks in each stripe group does not exceed the number of matched check blocks, which can significantly shorten the data recovery time, thereby improving data reliability and fault tolerance while ensuring the efficient operation of the system and data security.

[0094] The fault data recovery method provided in this application can be applied to Figure 1In the distributed storage system shown, the distributed storage system includes a distributed storage client, a storage service component and a storage management component, wherein the distributed storage client is responsible for interacting with the application or user, sending read and write requests, and receiving responses from the metadata server and the data server as well as the original data to be stored; the distributed storage client includes objects, blocks, virtual machines, containers and files. The storage service component is used to provide access interface services for the distributed storage client; specifically, the storage service component includes an object gateway service, a block device service and a file system service; the object gateway service is used to provide a RESTful API (a Web service interface design specification that follows the REST principle) interface compatible with a specific object storage system for the distributed storage client; the block device service allows the distributed storage client to read and write data in blocks; the file system service is used to provide an interface between the distributed storage client and the storage device, so that data can be stored and accessed in the form of files. As the core component of the distributed storage system, the storage management component provides distributed storage services in the form of objects. In the storage management component, data is stored as objects, each of which has a unique identifier and related data. The storage management component is responsible for distributing the objects on each storage node of the storage pool and providing data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access. The storage management component includes a storage pool, a metadata cluster and a monitoring service cluster. The storage pool consists of multiple storage nodes, each of which can contain multiple storage devices. The storage pool divides the storage resources to form different erasure storage pools to meet different storage needs. Specifically, the storage pool divides the original data of the distributed storage client through slicing and groups the objects, and finally stores each group in the underlying storage device. In the process of distributing the groups to the underlying storage device, the storage pool evenly distributes the data objects on each storage node of the storage pool through a distributed data balanced placement algorithm to avoid data hot spots and improve the performance of the distributed storage system. The metadata cluster is used to manage metadata in the distributed storage system. Metadata is data used to describe data, such as file attributes, access rights, storage location, etc. The metadata cluster stores this information in memory or disk to quickly respond to metadata requests from clients. The monitoring service cluster is used to monitor the status and performance of each component in the distributed storage system to ensure the stable operation of the system. The monitoring service cluster provides real-time information about the health status, performance bottlenecks, and potential problems of the distributed storage system by collecting and analyzing data.

[0095] Current correction coding technologies include correction coding based on Galois field operations. Galois field erasure codes are mainly Reed-Solomon (RS) codes, which are forward error-corrected channel codes. In distributed storage systems, RS codes have two parameters k and m, denoted as RS(k, m); where k represents the number of original data blocks and m represents the number of check blocks. For example, when the RS code is RS(8,2), it indicates that there are 8 original data blocks and 2 check blocks; the total number of data blocks n is the sum of the number of original data blocks k and the number of check blocks m, which is commonly expressed as n=k+m. When the redundancy level is n, n data blocks can be stored on object storage devices (OSDs) of n different storage nodes, so that m nodes can be tolerated to fail, and when no more than m storage nodes fail, only k data blocks need to be read from storage nodes that have not failed to reconstruct the data blocks in the failed node by calculation. The principle of RS code is to obtain a data vector by multiplying a vector D composed of k original data blocks by a generator matrix B. The data vector consists of k original data blocks and m check blocks. If a data block is lost, the lost data block can be restored through a series of calculations. It can be understood that the k value affects the cost of data recovery. The smaller the k value, the smaller the data dispersion, the larger the fault impact area, and the greater the cost of reconstructing data; the larger the k value, the more multiple data copies will increase the load on the distributed storage system network and disk, and the greater the cost of reconstructing data. The m value affects reliability and storage cost. The larger the m value, the higher the fault tolerance; the smaller the m value, the lower the data redundancy. Therefore, it is necessary to balance the two and solve the cost of reconstructing data for the distributed storage system.

[0096] In one embodiment, Figure 2 As shown, a method for recovering fault data is provided, which is applied to Figure 1 The distributed storage system in the example is used to illustrate, including the following steps:

[0097] S1: Get the original data.

[0098] It should be noted that original data refers to the data stored in the distributed storage system by users or applications through distributed storage clients.

[0099] S2: Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups.

[0100] It should be noted that the number of data blocks can be set according to actual needs, such as K+M, that is, the original data can be cut into K data blocks, and the corresponding M check blocks are generated by the erasure coding algorithm, where the data blocks are of the same size. The erasure coding algorithm (EC) is a commonly used data protection method, which divides the data into multiple fragments and generates redundant data blocks, namely check blocks, through encoding. The specific calculation process is not repeated here; the number of groups can also be set according to actual needs. For example, according to the best practices of the current distributed storage scale erasure pool, the number of groups can be set to K / 4, that is, K data blocks are divided into K / 4 groups, each group includes 4 data blocks, and the number of data blocks in each group is the same; striping refers to a method of dividing continuous data into data blocks of the same size and writing each segment of data to different disks.

[0101] S3: Perform erasure coding on the data blocks in the stripe grouping to generate a check block matching the data blocks in each stripe grouping.

[0102] It should be noted that the number of check blocks matching the data blocks in each stripe group can be set according to actual needs. The preferred value is M, that is, each stripe group corresponds to M check blocks, and the check blocks corresponding to each stripe group are also calculated and generated by the erasure code algorithm.

[0103] S4: Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks.

[0104] It should be noted that the health status of a data block can be determined by calculating a target logistic regression model. When the output value of the target logistic regression model is greater than a preset threshold, it indicates that the data block is abnormal and is an abnormal data block, also known as a sub-healthy data block.

[0105] S5: Adjust the abnormal data blocks between stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

[0106] It should be noted that adjusting the abnormal data blocks between stripe groups refers to adjusting the positions of the abnormal data blocks in each stripe group, that is, adjusting the abnormal data blocks in each stripe group that are larger than the number of check blocks to the stripe group with fewer abnormal data blocks, or merging the abnormal data blocks in each stripe group that are smaller than the number of check blocks to other stripe groups, so as to ensure that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group whose number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

[0107] S6: Update the check blocks that match the data blocks in each stripe group after adjusting the abnormal data blocks.

[0108] It should be noted that after the data blocks in each stripe group are adjusted, the check block corresponding to the stripe group needs to be recalculated so as to restore the data corresponding to the failed data block when a data block in the stripe group fails.

[0109] S7: In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0110] It should be noted that when a sub-healthy data block fails, the data of the failed data block is restored by decoding the non-abnormal data blocks and the check blocks in the stripe group.

[0111] In the above implementation, more efficient data management and storage are achieved by cutting the original data into multiple data blocks of equal size and performing striping grouping on these data blocks. The data blocks in each stripe group are encoded to generate check blocks. The increase of this redundant information provides a guarantee for subsequent data recovery. The health status of the data blocks in the stripe group is monitored to facilitate timely discovery of abnormal data blocks. After the abnormality is discovered, the distribution of the abnormal data blocks is adjusted to ensure that the number of abnormal data blocks in each stripe group does not exceed the number of matched check blocks. This can effectively prevent excessive abnormal data in a single stripe and ensure the overall recoverability of the data. This adjustment mechanism increases the system's tolerance for faults. The recovery of faulty data based on the updated check blocks can quickly locate the faulty data on the one hand, and quickly recover lost or damaged data blocks with the help of the check blocks on the other hand. The overall technical solution can significantly shorten the data recovery time, thereby improving data reliability and fault tolerance while ensuring efficient operation of the system and data security.

[0112] In some specific embodiments, cutting the original data into multiple data blocks includes: obtaining the data length and initial block length value of the original data; determining whether the length of the original data is an integer multiple of the initial block length value; in response to the length of the original data not being an integer multiple of the initial block length value, adding a placeholder at the end of the original data until the length of the original data is an integer multiple of the initial block length value; in response to the length of the original data being an integer multiple of the initial block length value, using an erasure code algorithm to cut the original data into multiple data blocks based on the initial block length value.

[0113] Specifically, the initial block length value can be set according to actual needs, and the preferred value is 4K. For example, if the data length of the original data is 47K, the initial block length value is 4K, and the data length of the original data is not an integer multiple of the data block length, that is, the original data will be divided into 11 initial blocks of 4K in size, and the remaining 3K will be classified as an incomplete initial block. At this time, 1K of zero data can be added to the end of the file so that the last initial data block also reaches 4K. In this embodiment, by adding a placeholder at the end of the original data so that its length becomes an integer multiple of the initial block length, it is ensured that the data can be evenly divided into data blocks of equal size, avoiding the occurrence of incomplete data blocks, improving the consistency of data processing and storage, and facilitating subsequent erasure code operations. The erasure code algorithm is used to cut the data into multiple data blocks of equal size, which not only ensures the uniform distribution of data blocks, but also can restore data through redundant information when a failure occurs.

[0114] In some specific embodiments, grouping multiple data blocks to obtain multiple stripe groups includes: grouping multiple data blocks according to a target grouping value to obtain multiple stripe groups, wherein each stripe group contains the same number of data blocks with equal data size and the same number of check blocks matching the data blocks, and the target grouping value is used to describe the number of data blocks contained in each stripe group.

[0115] Specifically, the target grouping value can be set according to the system load, and the present application does not impose any restrictions on this. It is preferably K / 4 as described above. For example, according to the 12+2 erasure code algorithm, the original data 47K is padded and divided into K=12 data blocks, and then the 12 data blocks are divided into 3 stripe groups according to the target grouping value K / 4, and each stripe group generates M=2 check blocks.

[0116] In the above implementation, by grouping data blocks and check blocks, more efficient storage and reading operations can be achieved, while improving the system's fault tolerance. When certain data blocks are damaged, the check blocks can be used for data recovery to ensure data integrity and reliability. In addition, evenly distributed data blocks and check blocks also help optimize system performance, reduce waste of storage resources, and improve storage efficiency.

[0117] In some specific implementations, the fault data recovery method further includes storing stripe groups and check blocks: storing the data blocks in the multiple stripe groups in different object storage devices respectively; storing the check blocks matching the data blocks in each stripe group in an object storage device different from the storage location of the data blocks in the multiple stripe groups; wherein the object storage devices are stored in different system storage nodes, wherein the object storage devices are OSD storage devices, i.e., disks. In this implementation, by storing the data blocks and check blocks in different storage nodes respectively, the fault tolerance and data security of the distributed storage system can be improved. If a node or storage device fails, the system can still recover the data through the check blocks on other nodes to ensure that the data is not lost. In addition, the decentralized storage of data and check blocks reduces the risk of single point failures, enhances the reliability and availability of the system, and also optimizes the performance of data access and improves the parallelism of data reading and writing.

[0118] In some specific implementations, obtaining the health status of data blocks in each stripe group includes: obtaining historical data of the object storage device and preprocessing the historical data, the historical data is used to describe the specifications and performance of the object storage device; extracting characteristic values ​​of the preprocessed historical data; dividing the characteristic values ​​of the historical data into a training set and a test set according to a preset ratio; selecting an initial logistic regression model; training the initial logistic regression model using the training set to obtain a first logistic regression model; adjusting the parameter value of the first logistic regression model to obtain a second logistic regression model; verifying and evaluating the second logistic regression model using the test set, and determining a target logistic regression model according to the evaluation result, wherein the expression of the target logistic regression model includes:

[0119]

[0120] Among them, m represents the time period, x i represents the probability of product failure in the i-th hour, f(x) represents the failure rate of the object storage device process, and p(x) represents the output value of the logistic regression model; according to the output value of the target logistic regression model, the health status of the data blocks in each stripe group is determined.

[0121] Specifically, the historical data of the object storage device may include information such as the hard disk model, manufacturer, usage time, temperature, read and write times, and fault records. The collected data is preprocessed, and the preprocessing steps include removing outliers, filling missing values, and data normalization. Corresponding features are extracted from the hard disk historical data, such as the total number of disks in the cluster, the read and write times, and the failure rate. The historical data feature values ​​are divided into a training set and a test set according to a preset ratio, where the preset ratio can be set according to actual needs, such as 7:3 or 8:2. After the data is divided, an initial logistic regression model is selected, and the selected model is trained using the training set of the historical data feature values, and the model parameters are adjusted to improve Prediction accuracy: Based on the test set, the trained model is evaluated using methods such as cross-validation. Evaluation indicators include accuracy and recall to verify the model's prediction ability and test the model's prediction accuracy for future hard disk failures. After the verification is completed, that is, when the evaluation result meets the preset standard, the target logistic regression model is obtained and deployed to the distributed storage system. The hard disk's operating status and prediction results are monitored in real time, and the model is retrained based on new data to regularly update the model and maintain its prediction accuracy. The preset standard can be that the accuracy reaches a certain preset accuracy value or the number of training times reaches a certain preset number of times. Both the preset accuracy value and the preset number of times can be set according to actual needs.

[0122] In the above implementation, the target logistic regression model is used to predict the failure probability value of the data block in the future time period, ensuring that the system can identify potential high failure rate data blocks in advance, so as to take preventive or repair measures in time to avoid data loss or system performance degradation, thereby improving the reliability and stability of the distributed storage system and optimizing the real-time monitoring and management of the health status of the data blocks.

[0123] In some specific embodiments, determining the health status of data blocks in each stripe grouping according to the output value of the target logistic regression model includes: comparing the output value of the target logistic regression model with a preset threshold; in response to the output value of the target logistic regression model being greater than the preset threshold, defining the data block corresponding to the output value of the target logistic regression model as an abnormal data block, wherein the preset threshold can be set according to actual needs, and the preferred value is 0.5. For example, assuming that the preset threshold is 0.5, the output value of the data block calculated by the target logistic regression model is 0.731, at this time, the output value is greater than the preset threshold, and the data block is defined as an abnormal data block; in this embodiment, by real-time monitoring and evaluation of the health status of the data blocks, the system can improve the overall data reliability and respond quickly when problems arise.

[0124] In some specific embodiments, the abnormal data blocks between stripe groups are adjusted: a process of adjusting the number of abnormal data blocks is performed: a plurality of stripe groups containing abnormal data blocks are obtained, and a second target stripe group is selected according to the number of abnormal data blocks in the plurality of stripe groups; the number of abnormal data blocks in the second target stripe group is compared with the number of check blocks that match the data blocks in the second target stripe group; in response to the number of abnormal data blocks in the second target stripe group being less than the number of check blocks that match the data blocks in the second target stripe group, any abnormal data block in the third target stripe group is exchanged with a normal data block in the second target stripe group until the number of abnormal data blocks contained in the second target stripe group is equal to the number of check blocks that match the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the third target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group; in response to the number of abnormal data blocks in the second target stripe group The number of normal data blocks is greater than the number of check blocks that match the data blocks in the second target stripe group, any abnormal data block in the second target stripe group is exchanged with the normal data block in the fourth target stripe group until the number of abnormal data blocks contained in the second target stripe group is equal to the number of check blocks that match the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the fourth target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group, and the number of abnormal data blocks in the fourth target stripe group is less than the number of check blocks that match the data blocks in the fourth target stripe group; the abnormal data block number adjustment process is repeated until the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group whose number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

[0125] Specifically, multiple stripe groups containing abnormal data blocks are obtained, and the stripe groups are sorted in descending order according to the number of abnormal data blocks in the multiple stripe groups. The second target stripe group is generally the stripe group with the highest sorting, that is, the second target stripe group has the largest number of abnormal data blocks, and the third target stripe group and the fourth target stripe group are both stripe groups ranked lower than the second target stripe group. For example, Figure 3As shown in the figure, in the distributed storage system, the original data is divided into 12 data blocks, namely D1-D12, and the corresponding check blocks are P1 and P2. The 12 data blocks are divided into three stripe groups A, B, and C. Stripe group A includes D1-D4 data blocks, stripe group B includes D5-D8 data blocks, and stripe group C includes D9-D12 data blocks. The corresponding check blocks generated by each stripe group are P11, P12, P21, P22, P31, and P32, that is, each stripe group contains 4 data blocks and 2 check blocks, each stripe can tolerate up to 2 abnormal data blocks without data loss. If there are 2 abnormal data blocks in stripe group A, 1 abnormal data block in stripe group B, and 1 abnormal data block in stripe group C, first detect stripe group A with the most abnormal data blocks, and determine that the number of abnormal data blocks in stripe group A is equal to the number of check blocks. At this time, the abnormal data blocks in stripe group A are not adjusted. There is 1 abnormal data block in both stripe group B and stripe group C. At this time, it is determined that they are both less than the number of check blocks, and the The abnormal data blocks in stripe group C are exchanged with the normal data blocks in stripe group B, or the abnormal data blocks in stripe group B are exchanged with the normal data blocks in stripe group C, so that there are no abnormal data blocks in stripe group B or stripe group C. If there are 3 abnormal data blocks in stripe group A, 2 abnormal data blocks in stripe group B, and 1 abnormal data block in stripe group C, then any abnormal data block in stripe group A is exchanged with any normal data block in stripe group C, and so on. Based on this, it can be ensured that under the condition that the number of abnormal data blocks in the stripe group does not exceed the corresponding number of check blocks, the abnormal data blocks are concentrated in one stripe group to improve data recovery efficiency. For example, when 4 data blocks of the 12+2 erasure code fail, the present application only needs to read the other 4 healthy data blocks for data recovery, while the current existing scheme needs to read 8 normal data blocks for data recovery. The recovery speed of the present application is increased by 2 times. Therefore, the present application shortens the data reading time and improves the data recovery efficiency based on this scheme.

[0126] In the above implementation, the fault tolerance and data repair efficiency of the distributed storage system are optimized by dynamically adjusting the number of abnormal data blocks in the stripe grouping. By judging whether the number of abnormal data blocks in the abnormal stripe grouping exceeds the fault tolerance threshold that matches the check block, and adjusting the abnormal data blocks to other stripe groups when necessary, it is ensured that the number of abnormal data blocks in each stripe grouping is always within the allowed range. This adjustment method effectively avoids the risk of data loss caused by too many abnormal data blocks in a single stripe grouping, further improves the stability and data reliability of the system, reduces redundant repair operations, and improves overall efficiency.

[0127] In some specific implementations, before adjusting the data blocks, the abnormal data blocks and corresponding storage node information in each stripe are obtained in the abnormal stripe grouping set, the storage node positions of the abnormal data blocks contained therein are recorded, and the data blocks are classified according to the storage node position range, such as dividing the data blocks stored on the same node into the first category, and the data blocks stored on different nodes into the second category, or dividing the data blocks stored within the preset range of the same node into the same category, and preferentially adjusting the data blocks in the storage nodes corresponding to the same category. When the number of normal data blocks in the corresponding nodes of the same category is less than the preset value, the corresponding abnormal data blocks are migrated to other nodes, and the abnormal data blocks are adjusted and data is recovered through the normal data blocks of other nodes, wherein the preset range and the preset value can be adjusted according to actual needs. This scheme reduces cross-node data transmission, reduces network bandwidth consumption and delay, and improves migration efficiency by preferentially migrating abnormal data blocks within the corresponding nodes of the same category. At the same time, if the corresponding nodes of the same category cannot meet the conditions, other nodes are selected for migration, which further optimizes the migration process, minimizes the impact on the overall performance of the system, and ensures the efficient distribution of data and the health of storage nodes.

[0128] In the above-mentioned fault data recovery method, the method includes: obtaining original data; cutting the original data into multiple data blocks, grouping the multiple data blocks to obtain multiple stripe groups; performing erasure code encoding on the data blocks in the stripe groups to generate check blocks that match the data blocks in each stripe group; obtaining the health status of the data blocks in each stripe group, and determining the abnormal data blocks according to the health status of the data blocks; adjusting the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing the abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing the abnormal data blocks is equal to the number of matched check blocks; updating the check blocks that match the data blocks in each stripe group after adjusting the abnormal data blocks; in response to the abnormal data blocks in the first target stripe group failing, recovering the data of the failed data blocks according to the check blocks corresponding to the updated first target stripe group; the present application By cutting the original data into multiple data blocks and grouping these data blocks in stripes, more efficient data management and storage can be achieved. The data blocks in each stripe group are encoded to generate check blocks. The increase in redundant information provides a guarantee for subsequent data recovery. The health status of the data blocks in the stripe group is monitored to facilitate timely detection of abnormal data blocks. After the abnormality is found, the distribution of abnormal data blocks is adjusted to ensure that the number of abnormal data blocks in each stripe group does not exceed the number of matched check blocks. This can effectively prevent excessive abnormal data in a single stripe and ensure the overall recoverability of the data. This adjustment mechanism increases the system's tolerance for faults. Faulty data is recovered based on updated check blocks. On the one hand, the faulty data can be quickly located, and on the other hand, the lost or damaged data blocks can be quickly recovered with the help of check blocks. The overall technical solution can significantly shorten the data recovery time, thereby improving data reliability and fault tolerance while ensuring the efficient operation of the system and data security.

[0129] It should be understood that although Figure 1-Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0130] In one embodiment, Figure 4As shown, a fault data recovery device is provided, comprising: a first acquisition module, a grouping module, an encoding module, a second acquisition module, an adjustment module, an update module and a data recovery module, wherein:

[0131] A first acquisition module is used to acquire original data;

[0132] A grouping module, used for cutting the original data into a plurality of data blocks, and grouping the plurality of data blocks to obtain a plurality of stripe groups;

[0133] An encoding module, used for performing erasure coding on the data blocks in the stripe grouping to generate a check block matching the data blocks in each stripe grouping;

[0134] A second acquisition module is used to acquire the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0135] An adjustment module, used for adjusting the abnormal data blocks between the stripe groups, so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0136] An updating module, used for updating the check blocks matched by the data blocks in each stripe group after adjusting the abnormal data blocks;

[0137] The data recovery module is used for, in response to a failure of an abnormal data block in the first target stripe group, recovering data of the failed data block according to a check block corresponding to the updated first target stripe group.

[0138] As a preferred implementation, in the embodiment of the present invention, the grouping module is specifically used for:

[0139] Get the data length and initial block length value of the original data;

[0140] Determine whether the length of the original data is an integer multiple of the initial block length value;

[0141] In response to the length of the original data not being an integer multiple of the initial block length value, adding a placeholder at the end of the original data until the length of the original data is an integer multiple of the initial block length value;

[0142] In response to the length of the original data being an integer multiple of the initial block length value, an erasure coding algorithm is used to cut the original data into a plurality of data blocks according to the initial block length value.

[0143] As a preferred implementation, in the embodiment of the present invention, the grouping module is further used for:

[0144] Multiple data blocks are grouped according to a target grouping value to obtain multiple stripe groups, wherein each stripe group contains the same number of data blocks with equal data size and the same number of check blocks matching the data blocks, and the target grouping value is used to describe the number of data blocks contained in each stripe group.

[0145] As a preferred implementation, in the embodiment of the present invention, the device further includes a storage module, and the storage module is specifically used for:

[0146] The data blocks in the plurality of stripe groups are stored in different object storage devices respectively;

[0147] storing the check blocks that match the data blocks in each stripe group in an object storage device that is different from the storage location of the data blocks in the plurality of stripe groups;

[0148] Among them, object storage devices are stored in different system storage nodes.

[0149] As a preferred implementation, in the embodiment of the present invention, the second acquisition module is specifically used for:

[0150] Obtain historical data of the object storage device and preprocess the historical data, which is used to describe the specifications and performance of the object storage device;

[0151] Extract the characteristic values ​​of the preprocessed historical data;

[0152] Divide the historical data feature values ​​into training sets and test sets according to the preset ratio;

[0153] Select the initial logistic regression model;

[0154] The initial logistic regression model is trained using the training set to obtain a first logistic regression model;

[0155] Adjusting the parameter values ​​of the first logistic regression model to obtain a second logistic regression model;

[0156] The second logistic regression model is verified and evaluated using the test set. According to the evaluation results, the target logistic regression model is determined. The expression of the target logistic regression model includes:

[0157]

[0158] Among them, m represents the time period, x i represents the probability of product failure in the i-th hour, f(x) represents the failure rate of the object storage device process, and p(x) represents the output value of the logistic regression model;

[0159] The health status of the data blocks in each stripe group is determined based on the output value of the target logistic regression model.

[0160] As a preferred implementation manner, in the embodiment of the present invention, the second acquisition module is further configured to:

[0161] Compare the output value of the target logistic regression model with the preset threshold;

[0162] In response to an output value of the target logistic regression model being greater than a preset threshold, a data block corresponding to the output value of the target logistic regression model is defined as an abnormal data block.

[0163] As a preferred implementation, in the embodiment of the present invention, the adjustment module is specifically used for:

[0164] Execute the abnormal data block number adjustment process:

[0165] Acquire multiple stripe groups containing abnormal data blocks, and select a second target stripe group according to the number of abnormal data blocks in the multiple stripe groups;

[0166] Compare the number of abnormal data blocks in the second target stripe group with the number of check blocks that match the data blocks in the second target stripe group:

[0167] In response to the number of abnormal data blocks in the second target stripe group being less than the number of check blocks matching the data blocks in the second target stripe group, exchanging any abnormal data block in the third target stripe group with a normal data block in the second target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks matching the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the third target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group;

[0168] In response to the number of abnormal data blocks in the second target stripe group being greater than the number of check blocks that match the data blocks in the second target stripe group, exchanging any abnormal data block in the second target stripe group with a normal data block in a fourth target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks that match the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the fourth target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group, and the number of abnormal data blocks in the fourth target stripe group is less than the number of check blocks that match the data blocks in the fourth target stripe group;

[0169] Repeat the process of adjusting the number of abnormal data blocks until the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group whose number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

[0170] For the specific definition of the fault data recovery device, please refer to the definition of the fault data recovery method above, which will not be repeated here. Each module in the above-mentioned fault data recovery device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0171] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a fault data recovery method is implemented.

[0172] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0174] S1: Get raw data;

[0175] S2: Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0176] S3: Perform erasure coding on the data blocks in the stripe group to generate check blocks that match the data blocks in each stripe group;

[0177] S4: Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0178] S5: Adjust the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0179] S6: updating and adjusting the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0180] S7: In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0181] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0182] S1: Get raw data;

[0183] S2: Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0184] S3: Perform erasure coding on the data blocks in the stripe group to generate check blocks that match the data blocks in each stripe group;

[0185] S4: Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0186] S5: Adjust the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0187] S6: updating and adjusting the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0188] S7: In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0189] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0190] S1: Get raw data;

[0191] S2: Cut the original data into multiple data blocks, group the multiple data blocks, and obtain multiple stripe groups;

[0192] S3: Perform erasure coding on the data blocks in the stripe group to generate check blocks that match the data blocks in each stripe group;

[0193] S4: Obtain the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks;

[0194] S5: Adjust the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks;

[0195] S6: updating and adjusting the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted;

[0196] S7: In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

[0197] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0198] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for recovering faulty data, characterized in that: include: Get the original data; Cutting the original data into multiple data blocks, and grouping the multiple data blocks to obtain multiple stripe groups; Performing erasure coding on the data blocks in the stripe groups to generate check blocks that match the data blocks in each stripe group; Acquire the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks; Adjust the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks; Update and adjust the check blocks that match the data blocks in each stripe group after the abnormal data blocks are adjusted; In response to a failure of an abnormal data block in the first target stripe group, data of the failed data block is restored according to a check block corresponding to the updated first target stripe group.

2. The method according to claim 1, characterized in that Cutting the original data into a plurality of data blocks comprises: Obtaining the data length and initial block length value of the original data; Determine whether the length of the original data is an integer multiple of the initial block length value; In response to the length of the original data not being an integer multiple of the initial block length value, adding a placeholder at the end of the original data until the length of the original data is an integer multiple of the initial block length value; In response to the length of the original data being an integer multiple of the initial block length value, an erasure coding algorithm is used to cut the original data into a plurality of data blocks according to the initial block length value.

3. The method according to claim 2, characterized in that Grouping the multiple data blocks to obtain multiple stripe groups includes: The multiple data blocks are grouped according to a target grouping value to obtain multiple stripe groups, wherein each stripe group contains the same number of data blocks with equal data size and the same number of check blocks matching the data blocks, and the target grouping value is used to describe the number of data blocks contained in each stripe group.

4. The method according to claim 3, characterized in that The method further comprises storing the stripe grouping and the check block: storing the data blocks in the plurality of stripe groups in different object storage devices respectively; storing the check blocks matching the data blocks in each stripe group to an object storage device different from the storage location of the data blocks in the plurality of stripe groups; Wherein, the object storage device is stored in different system storage nodes.

5. The method according to claim 1, characterized in that Obtaining the health status of the data blocks in each stripe group includes: Acquire historical data of the object storage device and preprocess the historical data, wherein the historical data is used to describe specifications and performance of the object storage device; Extract the characteristic values ​​of the preprocessed historical data; Dividing the historical data feature values ​​into a training set and a test set according to a preset ratio; Select the initial logistic regression model; Using the training set to train the initial logistic regression model to obtain a first logistic regression model; Adjusting the parameter values ​​of the first logistic regression model to obtain a second logistic regression model; The second logistic regression model is verified and evaluated using the test set, and a target logistic regression model is determined according to the evaluation result. The expression of the target logistic regression model includes: Among them, m represents the time period, x i represents the probability of product failure in the i-th hour, f(x) represents the failure rate of the object storage device process, and p(x) represents the output value of the logistic regression model; The health status of the data blocks in each stripe group is determined according to the output value of the target logistic regression model.

6. The method according to claim 5, characterized in that Determining the health status of data blocks in each stripe group according to the output value of the target logistic regression model includes: Comparing the output value of the target logistic regression model with a preset threshold; In response to an output value of the target logistic regression model being greater than a preset threshold, a data block corresponding to the output value of the target logistic regression model is defined as an abnormal data block.

7. The method according to claim 1, characterized in that Adjusting the abnormal data blocks between the stripe groups so that the number of abnormal data blocks in the stripe group containing the abnormal data blocks does not exceed the number of matched check blocks includes: Execute the abnormal data block number adjustment process: Acquire multiple stripe groups containing the abnormal data blocks, and select a second target stripe group according to the number of abnormal data blocks in the multiple stripe groups; Compare the number of abnormal data blocks in the second target stripe group with the number of check blocks that match the data blocks in the second target stripe group: In response to the number of abnormal data blocks in the second target stripe group being less than the number of check blocks matching the data blocks in the second target stripe group, exchanging any abnormal data block in the third target stripe group with a normal data block in the second target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks matching the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the third target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group; In response to the number of abnormal data blocks in the second target stripe group being greater than the number of check blocks matching the data blocks in the second target stripe group, exchanging any abnormal data block in the second target stripe group with a normal data block in a fourth target stripe group until the number of abnormal data blocks included in the second target stripe group is equal to the number of check blocks matching the data blocks in the second target stripe group, wherein the number of abnormal data blocks in the fourth target stripe group is less than or equal to the number of abnormal data blocks in the second target stripe group, and the number of abnormal data blocks in the fourth target stripe group is less than the number of check blocks matching the data blocks in the fourth target stripe group; Repeat the abnormal data block number adjustment process until the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group whose number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks.

8. A fault data recovery device, characterized in that: The device comprises: A first acquisition module is used to acquire original data; A grouping module, used for cutting the original data into a plurality of data blocks, and grouping the plurality of data blocks to obtain a plurality of stripe groups; An encoding module, used for performing erasure coding on the data blocks in the stripe grouping to generate a check block matching the data blocks in each stripe grouping; A second acquisition module is used to acquire the health status of the data blocks in each stripe group, and determine the abnormal data blocks according to the health status of the data blocks; An adjustment module, used for adjusting the abnormal data blocks between the stripe groups, so that the number of abnormal data blocks in the stripe groups containing abnormal data blocks is equal to the number of matched check blocks, or there is only one stripe group in which the number of abnormal data blocks is less than the number of matched check blocks, and the number of abnormal data blocks in the remaining stripe groups containing abnormal data blocks is equal to the number of matched check blocks; An updating module, used for updating the check blocks matched by the data blocks in each stripe group after adjusting the abnormal data blocks; The data recovery module is used for, in response to a failure of an abnormal data block in the first target stripe group, recovering data of the failed data block according to a check block corresponding to the updated first target stripe group.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images