Data disaster recovery system based on LRC

Abstract

The invention relates to the technical field of data processing, in particular to a data disaster recovery system based on LRC, which comprises a user file, a user interface layer and a data cluster, and is characterized in that the user interface layer compresses and packs the user file into a data packet and confirms the corresponding relationship between the user file and the data packet and the corresponding relationship between the data packet and the data cluster; the data packet is uploaded to a data packet processing node, the data processing node blocks the data packet to form data blocks and creates a mapping relation between the data packet and the data blocks, and the data processing node also cuts the data blocks into a plurality of data fragments; a fixed number of local check slices and global check slices are respectively generated by utilizing the data fragments in an LRC mode, mapping relations between the data blocks and the data fragments and between the local check slices and the global check slices are created, and the data fragments, the local check slices and the global check slices are sent to data nodes of different regions to be stored. The invention provides the data disaster recovery system with high security.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data disaster recovery system based on LRC. Background Art

[0002] The theory of distributed storage system appeared in the 1970s, but it has been widely used in the past 20 years or so. With the rise of the Internet, a huge amount of data is generated every day. The storage and management of this data has far exceeded the processing capacity of a single computer. Large Internet companies have developed various distributed storage systems according to their own business needs, including distributed block storage, distributed file systems, object storage, etc. The underlying implementation of these distributed storage systems generally uses multiple copies, erasure codes (EC) or EC's improved LRC (Local Repairable Codes). These storage systems have a common feature, which is that all replica data or data slices are stored in the same region. This is a good choice for data that does not require particularly high security, but for data with particularly high security requirements, the data may be lost in major disasters such as wars and earthquakes.

[0003] In order to cope with the consequences of such extreme disasters, the industry has developed a distributed storage system with two locations, three centers and five copies, which is very wasteful of resources. Its main purpose is data disaster recovery.

[0004] Now the network transmission speed is getting faster and faster, and data centers are spread across different regions. For example, the data centers (computing centers) currently being built by various operators, China Telecom proposed a 2+4+31+X data center layout in 2020, China Mobile proposed a 4+N+31+X data center layout, and China Unicom’s M+1+N data center layout. This fully demonstrates that data centers across different regions of the country have gradually become a reality.

[0005] If a piece of data is divided into multiple data shards (source data shards), each of which has the same size, and then a number of redundant data shards are calculated through relevant algorithms, the redundant data shards and source data shards are stored in data centers in different regions. If there are 32 source data shards and 4 redundant data shards, then these 36 data shards are stored in data centers in 36 regions. In this way, if a data center in a certain region encounters a serious disaster and the data center is completely destroyed, we can also use other data shards to calculate the lost data shards, so that a higher disaster recovery effect can be achieved at a lower cost. Because only a small amount of redundant data is used for a piece of data, the security of the data is greatly improved. Still using the above example, the ratio of storage capacity to data can reach 36 / 32=1.125, which means that the wasted storage capacity is very low. Summary of the invention

[0006] The purpose of the present invention is to provide a data disaster recovery system based on LRC, which intends to divide user data into blocks first, and then fragment the blocks. For data reliability and reconstruction efficiency, LRC (local repairable code) encoding is performed on the data fragments in each block of data to generate local verification fragments and global verification fragments. Finally, all fragments corresponding to each block (source fragments, local verification fragments, global fragments) are stored in each data center according to the rule of storing one fragment in one region, so as to solve the technical problems existing in the above-mentioned background technology.

[0007] To achieve the above object, the technical solution of the present invention is as follows:

[0008] A data disaster recovery system based on LRC comprises: user files, a user interface layer and a data cluster, wherein the user interface layer compresses and packages user files into data packets and confirms the corresponding relationship between user files and data packets and the corresponding relationship between data packets and data clusters, wherein the data cluster comprises: a data packet processing node and a plurality of data nodes stored in different regions, wherein the data packets are uploaded to the data packet processing node, the data processing node cuts the data packets into data blocks and creates a mapping relationship between the data packets and the data blocks, the data processing node also cuts the data blocks into a plurality of data slices, and respectively generates a fixed number of local check slices and global check slices by using the data slices in an LRC manner, creates a mapping relationship between data blocks and data slices, local check slices and global check slices, and sends the plurality of data slices, local check slices and global check slices to data nodes in different regions for storage.

[0009] Furthermore, when the user interface layer packages user files, if the user file size is smaller than the recommended value, multiple user files smaller than the recommended value are merged and packaged; if the user file size is larger than the recommended value, the single user file is compressed and packaged into a data packet.

[0010] Furthermore, it also includes: a database, the user files and data packets are stored in the database, and the user interface layer confirms the corresponding relationship between the user files and the data packets and the corresponding relationship between the data packets and the data clusters in the database.

[0011] Furthermore, the data block size is fixed. If the data packet size does not meet the block cutting standard, the data packet size is increased tenfold before block cutting.

[0012] Furthermore, the data processing node creates a mapping table of data packets and data blocks, and stores the mapping relationship between data packets and data blocks in the mapping table. The data blocks corresponding to the local check pieces and the global check pieces are the same as the data blocks corresponding to the data slices.

[0013] Furthermore, data nodes of different data clusters can be stored in one data center.

[0014] Furthermore, the data node saves the data slices in the form of files, and the file name of the data slice is composed of the hash value of the data slice, the position of the data block where the slice is located in the data packet, and the position of the data slice in the data block; each data packet creates a folder in the data node, and the data slices in the same data packet are saved in the folder named after the data packet; when the data packet is uploaded, the data of the folder is also determined, and the MD5 value md5-save of all the slices in the folder is calculated together and saved in the folder for later data maintenance.

[0015] Furthermore, the data node undertakes the functions of data integrity checking and data reconstruction.

[0016] Furthermore, the specific steps of the data integrity check are as follows:

[0017] Step 1: Read all data fragments in the data packet folder, and then calculate the total MD5 value md5-chk of all data;

[0018] Step 2: Compare the calculated total MD5 value md5-chk with the md5-save saved in the folder. If they are the same, it means that the data fragmentation data saved in the data packet is correct;

[0019] Step 3: If md5-chk is different from md5-save, it means that there are data errors in the data fragments in the data packet. You need to further check which data fragments have errors.

[0020] Step 4: Check the data integrity of the data shard. You only need to read the data of the data shard file, calculate its hash value, and compare the calculated hash value hash-chk with the hash value hash-save in the shard file name.

[0021] Step 5: If hash-save is the same as hash-chk, the data in the data shard file is correct. If they are different, the file name of the wrong data shard needs to be reported to the data packet processing node;

[0022] Step 6: The data packet processing node queries the data shard-data block mapping table, and sends the data block information of the error shard to the data node where the error data shard is located;

[0023] Step 7: After the data node obtains the data block information, it will obtain other data shards, local check shards, and global check shards in the data block based on the LRC principle to reconstruct the erroneous data shards.

[0024] Furthermore, the data reconstruction is specifically as follows: after detecting that an error occurs in the data fragmentation data, the data packet processing node will send the data block information of the erroneous fragmentation to the data node for data reconstruction. During data reconstruction, local verification is performed first. If the reconstruction fails through the local method, global verification is used for reconstruction.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] Storing different data shards in the same data block in data nodes in different regions can avoid data loss when a devastating disaster occurs in a data center in a certain region; by storing data blocks in LRC shards, the waste of storage space can also be reduced, which solves the problem of disaster recovery and reduces costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0028] Figure 2 It is the file-data packet mapping table of the present invention;

[0029] Figure 3 It is the data block-data packet mapping table of the present invention;

[0030] Figure 4 It is the data fragmentation-data block mapping table in the present invention;

[0031] Figure 5 is a flow chart of data reconstruction in the present invention;

[0032] Figure 6 It is a schematic diagram of data packet cutting in the present invention;

[0033] Figure 7 It is a schematic diagram of data block slicing in the present invention;

[0034] Figure 8 It is a schematic diagram of data fragmentation in a data packet folder in the present invention;

[0035] Fig. 9 It is a flow chart of file uploading of the present invention;

[0036] Fig.10 It is a file downloading flow chart of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0038] See also Figures 1 to 10 As shown, a data disaster recovery system based on LRC includes: user files, a user interface layer and a data cluster, the user interface layer is the client of the distributed disaster recovery system; the user interface layer compresses and packages the user files into data packets and confirms the corresponding relationship between the user files and the data packets and the corresponding relationship between the data packets and the data cluster, the data cluster includes: a data packet processing node and a number of data nodes stored in different regions, the data packet is uploaded to the data packet processing node, the data processing node cuts the data packet into data blocks and creates a mapping relationship between the data packets and the data blocks, the data processing node also cuts the data blocks into a number of data slices, each data block generates a fixed number of data slices (for example, 64); and the data slices are used to generate a fixed number of local check slices (assuming 4) and global check slices (assuming 4) respectively through the LRC method, create a mapping relationship between data blocks and data slices, local check slices and global check slices, and save the mapping relationship between data blocks and slices in a mapping table (the data blocks corresponding to the local check slices and the global check slices are the same as the data blocks corresponding to the data slices) as shown in FIG. Figure 4 As shown, Slice: saves the name of the slice, which is a combination of the hash and the location of the data block where the slice is located; DataBlock: Same as above, saves the datablock to which the slice belongs, and saves the datablock name; DataNode: the name of the data node saved by the slice, and the correspondence between the data node name and the data node IP and region is determined during deployment; several data slices, local check slices, and global check slices are sent to data nodes in different regions for storage, and slices of the same data block are preferably stored on different data nodes. In this way, data upload and storage are completed.

[0039] The user interface layer will compress and package multiple files of the user. The size depends on the actual situation. When the user interface layer packages the user files, if the user file size is smaller than the recommended value, multiple user files smaller than the recommended value will be merged and packaged; if the user file size is larger than the recommended value, the single user file will be compressed and packaged into a data packet; for example: if the user's file is a small file, the file will be merged and packaged into a relatively larger data packet, such as the recommended value of 20G. If the user's file is large and a single file exceeds 20G, this file will be compressed and packaged into a data packet.

[0040] The data disaster recovery system based on LRC also includes: a database, user files and data packets are stored in the database, and the user interface layer confirms the corresponding relationship between user files and data packets and the corresponding relationship between data packets and data clusters in the database, such as Figure 2As shown; when a user queries or downloads a file, he needs to first query the database for the correspondence between the file and the data packet, as well as the corresponding cluster, and then send the data packet information to the corresponding data packet processing node to extract the data packet. After the user interface layer gets the data packet, it decompresses the data packet to obtain the relevant file.

[0041] The data block size is fixed, for example: 512M; if the data packet size does not meet the block cutting standard, the data packet size will be increased tenfold before block cutting, that is, a 0 is added to the last digit of the data packet size value.

[0042] The data processing node creates a mapping table of data packets and data blocks, such as Figure 3 As shown, the mapping relationship between data packets and data blocks is stored in a mapping table.

[0043] In the mapping table:

[0044] packagename is the name of the data package. A data package can be regarded as a package file formed by compressing and packaging multiple user files. The name of the data package is the md5 value of the data package.

[0045] Datablock: data block name, which is composed of the hash of the data block and the position of the data block in the data packet;

[0046] Block-Slice tab: Data block and slice mapping table. Because the amount of data cut is large, all data is stored in one data table, which will be large and inconvenient to query. Therefore, the data block and slice mapping table needs to be divided into separate tables;

[0047] File download process:

[0048] To download a file:

[0049] 1. The user first queries the file-packet mapping table at the user interface layer ( Figure 2 ), obtain the data packet and data packet processing node corresponding to the file, and send the download request to the data packet processing node.

[0050] 2. The data packet processing node queries the data block-data packet mapping table in the data packet processing node according to the data packet request to be downloaded ( Figure 3 ), obtain all data blocks corresponding to the data packet, and traverse the data blocks corresponding to the data packet;

[0051] 3. The data packet processing node obtains the data node where each data shard of the data block is located from the corresponding data shard-data block table according to the information obtained from the data block-data packet mapping table, and sends a request to download the data shard to the data node corresponding to the data shard;

[0052] 4. The data node sends the data shard data to the data packet processor according to the data shard download request;

[0053] 5. The packet processor receives each data fragment. During the fragment reception process, the remaining fragments in the data block are calculated according to the LRC method. If all the fragments in the data block have been obtained, the processing of the data block is stopped and the generated data block is sent to the user interface layer node.

[0054] 6. In the user data interface layer, if all the data blocks that the data packet depends on have been obtained, the data packet is downloaded successfully;

[0055] 7. The user data interface layer decompresses the corresponding data packets to obtain the files required by the user;

[0056] The data nodes of the cluster are stored in various regions. It is best to deploy a data node in the cluster in the data center of each region. Data nodes of different clusters can be stored in one data center.

[0057] The data node saves the data shards in the form of files. The file name of the data shard is composed of the hash value of the data shard, the position of the data block where the shard is located in the data packet, and the position of the data shard in the data block. Each data packet creates a folder in the data node, and the data shards in the same data packet are saved in a folder named after the data packet. When the data packet is uploaded, the data in the folder is also determined, and the MD5 value md5-save is calculated for all the shards in the folder together and saved in the folder for later data maintenance.

[0058] Data nodes are responsible for data integrity checking and data reconstruction.

[0059] The specific steps of data integrity check are as follows:

[0060] Step 1: Read all data fragments in the data packet folder, and then calculate the total MD5 value md5-chk of all data;

[0061] Step 2: Compare the calculated total MD5 value md5-chk with the md5-save saved in the folder. If they are the same, it means that the data fragmentation data saved in the data packet is correct;

[0062] Step 3: If md5-chk is different from md5-save, it means that there are data errors in the data fragments in the data packet. You need to further check which data fragments have errors.

[0063] Step 4: Check the data integrity of the data shard. You only need to read the data of the data shard file, calculate its hash value, and compare the calculated hash value hash-chk with the hash value hash-save in the shard file name.

[0064] Step 5: If hash-save is the same as hash-chk, the data in the data shard file is correct. If they are different, the file name of the wrong data shard needs to be reported to the data packet processing node;

[0065] Step 6: The data packet processing node queries the data shard-data block mapping table, and sends the data block information of the error shard to the data node where the error data shard is located;

[0066] Step 7: After the data node obtains the data block information, it will obtain other data shards, local check shards, and global check shards in the data block based on the LRC principle to reconstruct the erroneous data shards.

[0067] Data reconstruction is specifically as follows: after detecting errors in data shards, the data packet processing node will send the data block information of the erroneous shards to the data node for data reconstruction. During data reconstruction, local verification is given priority for reconstruction. If reconstruction fails through local methods, global verification is used for reconstruction.

[0068] It is necessary to add that:

[0069] 1. The size of the data block, the number of data slices, the local verification method and number, and the number of global verification slices should be designed in multiple typical combinations according to relevant actual scenarios during system development, and selected when deploying the cluster; after these deployment parameters are set, they will not be allowed to be modified in the future;

[0070] 2. If Figure 6 As shown in the figure, when the data packet is cut into blocks, the size of the data block is fixed (for example, 512M), and the number of data blocks varies according to the size of the data packet. The sequence number of the data block is the position of the data block in the data packet, for example, B0 is the first 512M data, B1 is the second 512M data, and so on;

[0071] 3. The number of data block slices varies according to the data block size and data shards. When deploying a cluster, you need to select relevant cluster parameters. If the selected parameter combination is: data block size is 512M, the number of data shards is 64 (data shard size is 8M), local verification uses row verification, and the row verification shards are 4, that is, one row verification shard is generated for every 16 data shards, and the global verification shards are also 4, indicating that the data redundancy is 4, that is, if 4 data shards are lost in each data block, the data shards can be restored. Figure 7 As shown, there will be the following relationship:

[0072] The relationship between data row local check sharding and data sharding:

[0073]

[0074] Right now:

[0075] A0s0+A1s1+...+A15s15=L0 ...

[0077] A0s48+A1s49+...+A15s63=L3

[0078] A0~A15 is a group of non-zero coefficients.

[0079] The relationship between global verification sharding and data sharding:

[0080]

[0081] It is a linearly independent coefficient matrix, which needs to be set first. In this way, if 4 shards are discarded at the same time, the lost shard data can be recovered.

[0082] The purpose of introducing row checksum is to speed up data reconstruction when only one shard in a row is lost, so that data can be restored using fewer data shards.

[0083] When a data block has a large number of data shards, row and column checks can be introduced at the same time. If the data redundancy needs to be improved, the number of global check shards can be increased. The more check data there is, the larger the coefficient matrix will be, and the more difficult it will be to find a linearly independent coefficient matrix.

[0084] In actual implementation, the local check slices and the global check slices are numbered uniformly. For the above example, the local row check slices can be set to 64 to 67, and the global check slices can be set to 68 to 71. This will make it more convenient in subsequent data reconstruction.

[0085] In this way, different shards of the same data block are stored on data nodes in data centers in different regions. When a data center in a certain region is catastrophically damaged, the data will not be lost.

[0086] 4. As mentioned before, the data shards in the data node are saved in the form of files in a folder named after the data packet. The naming method of the data shard file is the hash value of the data shard (slicehash) + the position offset of the data block to which the shard belongs in the data packet + the position offset of the shard in the data block. For example, sliceehash_6_70, sliceehash is the hash value of the data shard, 6 is the offset of the data block corresponding to the shard in the data packet, and 70 is the offset of the data shard in the data block. For the above example, the data shard 70 indicates that the shard is a global checksum shard.

[0087] The data packet folder in the data node will save the md5 value of all data fragments in the folder when they are merged together, such as Figure 8 The md5_5 shown in the figure, where the 5 at the end means that the data packet has 5 fragments stored in the data node, and md5_5 stores the md5 value when the 5 data fragments are merged together.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data disaster recovery system based on LRC, characterized in that: include: User files, user interface layer and data cluster, the user interface layer compresses and packages user files into data packets and confirms the correspondence between user files and data packets and the correspondence between data packets and data clusters, the data cluster includes: a data packet processing node and several data nodes stored in different regions, the data packets are uploaded to the data packet processing node, the data processing node cuts the data packets into data blocks and creates a mapping relationship between data packets and data blocks, the data processing node also cuts the data blocks into several data slices, and uses the data slices to generate a fixed number of local verification slices and global verification slices respectively through the LRC method, creates a mapping relationship between data blocks and data slices, local verification slices and global verification slices, and sends several data slices, local verification slices and global verification slices to data nodes in different regions for storage.

2. The data disaster recovery system based on LRC according to claim 1, characterized in that: When the user interface layer packages user files, if the user file size is smaller than the recommended value, multiple user files smaller than the recommended value are merged and packaged; if the user file size is larger than the recommended value, the single user file is compressed and packaged into a data packet.

3. The data disaster recovery system based on LRC according to claim 1, characterized in that: Also includes: A database, wherein the user files and data packets are stored in the database, and the user interface layer confirms the corresponding relationship between the user files and the data packets and the corresponding relationship between the data packets and the data clusters in the database.

4. The data disaster recovery system based on LRC according to claim 1, characterized in that: The data block size is fixed. If the data packet size does not meet the block cutting standard, the data packet size will be increased tenfold before block cutting.

5. The data disaster recovery system based on LRC according to claim 1, characterized in that: The data processing node creates a mapping table of data packets and data blocks, and stores the mapping relationship between data packets and data blocks in the mapping table. The data blocks corresponding to the local check pieces and the global check pieces are the same as the data blocks corresponding to the data slices.

6. The data disaster recovery system based on LRC according to claim 1, characterized in that: Data nodes of different data clusters can be stored in one data center.

7. The data disaster recovery system based on LRC according to claim 1, characterized in that: The data node saves the data slices in the form of files, and the file name of the data slice is composed of the hash value of the data slice, the position of the data block where the slice is located in the data packet, and the position of the data slice in the data block; each data packet creates a folder in the data node, and the data slices in the same data packet are saved in the folder named after the data packet; when the data packet is uploaded, the data of the folder is also determined, and the MD5 value md5-save is calculated for all the slices in the folder together, and saved in the folder for later data maintenance.

8. The data disaster recovery system based on LRC according to claim 1, characterized in that: The data node is responsible for data integrity checking and data reconstruction.

9. The data disaster recovery system based on LRC according to claim 8, characterized in that: The specific steps of the data integrity check are as follows: Step 1: Read all data fragments in the data packet folder, and then calculate the total MD5 value md5-chk of all data; Step 2: Compare the calculated total MD5 value md5-chk with the md5-save saved in the folder. If they are the same, it means that the data fragmentation data saved in the data packet is correct; Step 3: If md5-chk is different from md5-save, it means that there are data errors in the data fragments in the data packet. You need to further check which data fragments have errors. Step 4: Check the data integrity of the data shard. You only need to read the data of the data shard file, calculate its hash value, and compare the calculated hash value hash-chk with the hash value hash-save in the shard file name. Step 5: If hash-save is the same as hash-chk, the data in the data shard file is correct. If they are different, the file name of the wrong data shard needs to be reported to the data packet processing node; Step 6: The data packet processing node queries the data shard-data block mapping table, and sends the data block information of the error shard to the data node where the error data shard is located; Step 7: After the data node obtains the data block information, it will obtain other data shards, local check shards, and global check shards in the data block based on the LRC principle to reconstruct the erroneous data shards.

10. A data disaster recovery system based on LRC according to claim 8 or 9, characterized in that: The data reconstruction is specifically as follows: after detecting that an error occurs in the data fragmentation data, the data packet processing node will send the data block information of the error fragmentation to the data node for data reconstruction. During data reconstruction, local verification is performed first. If the reconstruction fails through the local method, global verification is used for reconstruction.

Images