Data decentralization storage and disaster recovery processing method, device and storage medium

Abstract

The application discloses a kind of data decentralized storage and disaster recovery processing method, equipment and storage medium capable of guaranteeing the stability of own data and intelligent disaster recovery.The processing method comprises the following steps:Step 100, m nodes in n nodes are obtained to form a storage group, the storage group is divided into m1 data nodes and m2 redundant nodes, m < n, m1+m2=m;Step 200, the data of the n+1th node is received, and the data is stored in the appropriate data node according to the BFT consensus;Step 300, each node in the storage group verifies its own data and sends the verification result to other nodes in the storage group;Step 400, all nodes in the storage group receive the verification result and generate the BFT consensus in the group, and then determine whether there is a malicious node;If so, the malicious node is removed, and then step Step 500 is entered;Step 500, select a new node that meets the BFT consensus from the remaining n-m nodes to join the storage group, forming a new storage group;Step 600, the new storage group re-forms the BFT consensus in the group.

Description

Technical Field

[0001] This invention relates to the technical field of data storage and disaster recovery, and more specifically, to methods, devices, and storage media for decentralized data storage and disaster recovery. Background Technology

[0002] As a crucial management platform, the security and stability of storage systems are always the goals pursued by vendors and important indicators for customers. However, an increasing number of customers are placing higher demands on storage system management software, with intelligent disaster recovery being a key component. The essence of disaster recovery should be prevention; only by ensuring the stability of the data within the storage system itself can storage nodes be better managed, thereby ensuring greater stability for user businesses.

[0003] However, the most common disaster recovery management method for storage system data is manual backup and recovery. Manual backup refers to using multiple storage devices to back up data and prevent it from becoming unusable after abnormal deletion. However, this method consumes additional storage resources and cannot achieve real-time backup. Recovery works by saving the system index of deleted files and then reversing the process to recover the abnormally deleted files. For example, using the ext4 file system (Fourth extended file system) for data disaster recovery. Although this method saves the file inode information in the journal log when deleted, the system index of deleted files is often recycled and reused in the security field, so the probability of successfully recovering the files is often not high.

[0004] Data storage methods can be divided into centralized proprietary storage and decentralized storage. Decentralized storage is a new type of data storage method with advantages such as fast download speeds, high security, and low cost. Therefore, traditional centralized proprietary data storage services are being replaced by decentralized distributed open data storage services due to the difficulty in expanding and migrating data files, the vulnerability of highly centralized data to information tampering, and privacy leaks. However, existing decentralized storage methods have relatively complex storage processes, low reliability, or high space consumption, and are difficult to verify malicious nodes, which also poses a certain threat to data security.

[0005] It is evident that ensuring the stability of data in storage systems, especially decentralized storage systems, and achieving intelligent disaster recovery management of their own data are problems that need to be solved by those skilled in the art. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a data decentralized storage and disaster recovery processing method, device and storage medium that can ensure the stability of the data of the decentralized storage system itself and provide intelligent disaster recovery.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a method for decentralized data storage and disaster recovery is provided, the technical solution of which is as follows:

[0008] The methods for decentralized data storage and disaster recovery include the following steps:

[0009] Step 100: Obtain m nodes that can form BFT consensus from n nodes to form a storage group. Divide the storage group into m1 data nodes and m2 redundant nodes. The data nodes store the original data, and the redundant nodes store redundant data. m < n, m1 + m2 = m.

[0010] Step 200: Receive data from the (n+1)th node and store the data in the appropriate data node according to the BFT consensus.

[0011] Step 300: Each node in the storage group verifies its own data and sends the verification result to other nodes in the storage group.

[0012] Step 400: All nodes in the storage group receive the verification results and generate the group's BFT consensus. Then, it is determined whether there are any malicious nodes. If so, the malicious nodes are removed, and then the process proceeds to Step 500.

[0013] Step 500: Select new nodes that satisfy BFT consensus from the remaining nm nodes of n nodes and add them to the storage group to form a new storage group;

[0014] Step 600: The new storage group re-establishes the intra-group BFT consensus.

[0015] As a further improvement to the first aspect of the present invention: the algorithm for forming BFT consensus is PBFT, SBFT, HoneyBadger BFT, Hot Stuff, or BFT-SMaRt; the data of redundant nodes is determined by an erasure coding algorithm, wherein the erasure coding algorithm is XOR or RS.

[0016] As a further improvement to the first aspect of the invention: in Step 300, each node in the storage group verifies its own data according to the requirements of zero-knowledge proof.

[0017] As a further improvement of the first aspect of the present invention: the verification result of each data node is evidence that it has correctly and effectively stored the corresponding original data; the verification result of each redundant node is evidence that it has correctly and effectively calculated and stored the corresponding redundant data.

[0018] As a further improvement to the first aspect of the present invention: the algorithm that satisfies the requirements of zero-knowledge proof is zkSNARK, zkSTARKs, BulletProofs, or Groth16.

[0019] As a further improvement of the first aspect of the present invention: the malicious node is a data node that does not correctly store the original data, or a redundant node that does not correctly calculate the data, or an inactive node that does not provide verification results or cannot respond to the BFT consensus in a timely manner.

[0020] As a further improvement to the first aspect of the present invention: a node is removed only when the number of times it fails to provide a verification result or fails to respond to the BFT consensus in a timely manner reaches a preset value.

[0021] As a further improvement to the first aspect of the present invention: Step 600 includes:

[0022] Step 610: The new node calculates the lost data in the new storage group based on the erasure coding algorithm, and then recovers the lost data;

[0023] Step 620: The new node verifies its own data and sends the verification result to other nodes in the new storage group;

[0024] Step 630: Other nodes in the new storage group verify the verification result. If the verification passes, a BFT consensus is formed within the group. If the verification fails, the new node is removed, and the process returns to Step 500.

[0025] To achieve the above objectives, according to a second aspect of the present invention, a computer device is provided, the technical solution of which is as follows:

[0026] A computer device includes: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the data decentralized storage and disaster recovery processing method described in the first aspect above.

[0027] To achieve the above objectives, according to a third aspect of the present invention, a computer-readable storage medium is provided, the technical solution of which is as follows:

[0028] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data decentralized storage and disaster recovery processing method described in the first aspect above.

[0029] As can be seen, the data decentralized storage and disaster recovery processing method, device and storage medium of the present invention selects storage groups to store data through BFT consensus, realizes decentralized data storage, resists Byzantine attacks, significantly reduces storage costs and significantly improves reliability, and the storage group can automatically verify the stored data to achieve intelligent disaster recovery, effectively ensuring the stability of its own data and has strong practicality.

[0030] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] The accompanying drawings, which form part of this invention, are used to aid in understanding the invention. The content provided in the drawings and their related descriptions can be used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0032] Figure 1 This is a flowchart illustrating the first embodiment of the data decentralized storage and disaster recovery processing method of the present invention.

[0033] Figure 2 This is a flowchart illustrating step 600 in the first embodiment of the data decentralized storage and disaster recovery processing method of the present invention. Detailed Implementation

[0034] The present invention will now be clearly and completely described in conjunction with the accompanying drawings. Those skilled in the art will be able to implement the present invention based on these descriptions. Before describing the present invention in conjunction with the accompanying drawings, it should be particularly noted that:

[0035] The technical solutions and features provided in the various parts of this invention, including the following description, can be combined with each other without conflict.

[0036] Furthermore, the embodiments of the present invention described below are generally only some, not all, of the embodiments of the present invention. Therefore, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0037] Regarding the terminology and units used in this invention: The terms "comprising," "having," and any variations thereof in the specification, claims, and related parts of this invention are intended to cover non-exclusive inclusion.

[0038] Figure 1 This is a flowchart illustrating the first embodiment of the data decentralized storage and disaster recovery processing method of the present invention.

[0039] like Figure 1 As shown, the methods for decentralized data storage and disaster recovery include the following steps:

[0040] Step 100: Obtain m nodes that can form BFT consensus from n nodes to form a storage group. Divide the storage group into m1 data nodes and m2 redundant nodes. The data nodes store the original data, and the redundant nodes store redundant data. m < n, m1 + m2 = m.

[0041] Step 200: Receive data from the (n+1)th node and store the data in the appropriate data node according to the BFT consensus.

[0042] Step 300: Each node in the storage group verifies its own data and sends the verification result to other nodes in the storage group.

[0043] Step 400: All nodes in the storage group receive the verification results and generate the group's BFT consensus. Then, it is determined whether there are any malicious nodes. If so, the malicious nodes are removed, and then the process proceeds to Step 500.

[0044] Step 500: Select new nodes that satisfy BFT consensus from the remaining nm nodes of n nodes and add them to the storage group to form a new storage group;

[0045] Step 600: The new storage group re-establishes the intra-group BFT consensus.

[0046] In Step 100, the algorithm used to form the BFT (Byzantine Fault Tolerance) consensus can be, but is not limited to, PBFT (Practical Byzantine Fault Tolerance), SBFT (from the paper "SBFT: a Scalable and Decentralized Trust Infrastructure"), Honey Badger BFT, Hot Stuff, or BFT-SMaRt. Redundant node data is determined using an erasure coding algorithm, which can be, but is not limited to, XOR (exclusive OR) or RS (Reed-Solomon). According to the requirements of BFT consensus, m should be at least 4. However, in decentralized storage systems, the reliability of a single node is low. Therefore, in this invention, to ensure the reliability of decentralized storage and disaster recovery, the value of m is as large as possible, for example, ≥5. Furthermore, the larger the ratio of m² / m, the better, preferably ≥0.5.

[0047] In Step 300, each node in the storage group verifies its own data according to the requirements of zero-knowledge proof. Specifically, the verification result of each data node is evidence that it has correctly and effectively stored the corresponding original data, and the verification result of each redundant node is evidence that it has correctly and effectively calculated and stored the corresponding redundant data. The algorithms that meet the requirements of zero-knowledge proof are zkSNARK (zero-knowledge Succinct Non-Interactive Arguments of Knowledge), zkSTARKs (zero-knowledge Succinct Transparent Arguments of Knowledge), BulletProofs, or Groth16.

[0048] In Step 400, the malicious node is a data node that does not correctly store the original data, or a redundant node that does not correctly calculate the data.

[0049] Figure 2 This is a flowchart illustrating step 600 in the first embodiment of the data decentralized storage and disaster recovery processing method of the present invention.

[0050] like Figure 2 As shown, step 600 includes:

[0051] Step 610: The new node calculates the lost data in the new storage group based on the erasure coding algorithm, and then recovers the lost data;

[0052] Step 620: The new node verifies its own data and sends the verification result to other nodes in the new storage group;

[0053] Step 630: Other nodes in the new storage group verify the verification result. If the verification passes, a BFT consensus is formed within the group. If the verification fails, the new node is removed, and the process returns to Step 500.

[0054] Furthermore, the present invention also provides a second embodiment of a method for decentralized data storage and disaster recovery. Based on the first embodiment, the malicious node in the second embodiment further includes inactive nodes that fail to provide verification results or cannot respond to BFT consensus in a timely manner. To improve reliability, a node is only removed when the number of times it fails to provide verification results or respond to BFT consensus in a timely manner reaches a preset value. This further improves data stability and disaster recovery capabilities.

[0055] Figure 3 This is a schematic block diagram illustrating an embodiment of the computer device of the present invention.

[0056] like Figure 3 As shown, the computer device includes a processor 120 and a memory 110, with the memory 110 storing a computer program 111. When the computer program 111 is executed by the processor 120, the computer device can perform the steps of the data decentralized storage and disaster recovery processing method as described in the above method embodiment. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0057] The computer devices provided in the embodiments of the present invention may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0058] The processor 120 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 120 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 120 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 120 may further integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 120 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0059] The memory 110 may include one or more computer-readable storage media, which may be non-transitory. The memory 110 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 110 is used to store at least the following computer program 111, which, after being loaded and executed by the processor 120, is capable of implementing the relevant steps of the data decentralized storage and disaster recovery processing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 110 may also include an operating system 112 and data 113, and the storage method may be temporary storage or permanent storage. The operating system 112 may include Windows, Unix, Linux, etc. The data 113 may include, but is not limited to, accessed data, hot data, etc.

[0060] In some embodiments, the computer device may further include a display screen 130, an input / output interface 140, a communication interface 150, a power supply 160, and a communication bus 170.

[0061] Those skilled in the art will understand that Figure 3 The structures shown do not constitute a limitation on computer equipment and may include more or fewer components than illustrated.

[0062] It is understood that if the data decentralized storage and disaster recovery processing methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, magnetic disks, or optical disks, and other media capable of storing program code.

[0063] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the above-described data decentralized storage and disaster recovery processing method. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0064] The functions of each functional module of the computer-readable storage medium described in the embodiments of the present invention can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0065] The foregoing has provided a detailed description of the data decentralized storage and disaster recovery processing method, computer equipment, and storage medium provided by the embodiments of the present invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0066] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0068] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0069] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific implementations of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0070] The foregoing has described the relevant content of the present invention. Specific examples have been used in this specification to illustrate the principles and implementation methods of the present invention. The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. Methods for decentralized data storage and disaster recovery, including the following steps: Step 100: Obtain m nodes that can form BFT consensus from n nodes to form a storage group. Divide the storage group into m1 data nodes and m2 redundant nodes. The data nodes store the original data, and the redundant nodes store redundant data. m < n, m1 + m2 = m. Step 200: Receive data from the (n+1)th node and store the data in the appropriate data node according to the BFT consensus. Step 300: Each node in the storage group verifies its own data and sends the verification result to other nodes in the storage group. Step 400: All nodes in the storage group receive the verification results and generate the group's BFT consensus, and then determine whether there are any malicious nodes. If so, remove the malicious node and then proceed to step 500; Step 500: Select new nodes that satisfy BFT consensus from the remaining nm nodes of n nodes and add them to the storage group to form a new storage group; Step 600: The new storage group re-establishes intra-group BFT consensus; Step 600 includes: Step 610: The new node calculates the lost data in the new storage group based on the erasure coding algorithm, and then recovers the lost data; Step 620: The new node verifies its own data and sends the verification result to other nodes in the new storage group; Step 630: Other nodes in the new storage group verify the verification result. If the verification passes, a BFT consensus is formed within the group. If the verification fails, the new node is removed, and the process returns to Step 500.

2. The data decentralized storage and disaster recovery processing method as described in claim 1, characterized in that: The algorithm for forming BFT consensus is PBFT, SBFT, Honey Badger BFT, Hot Stuff, or BFT-SMaRt; the data of redundant nodes is determined by erasure coding algorithm, which is XOR or RS.

3. The data decentralized storage and disaster recovery processing method as described in claim 1, characterized in that: In Step 300, each node in the storage group verifies its own data according to the requirements of zero-knowledge proof.

4. The data decentralized storage and disaster recovery processing method as described in claim 3, characterized in that: The verification result of each data node serves as evidence that it has correctly and effectively stored the corresponding original data; the verification result of each redundant node serves as evidence that it has correctly and effectively calculated and stored the corresponding redundant data.

5. The data decentralized storage and disaster recovery processing method as described in claim 3, characterized in that: Algorithms that meet the requirements of zero-knowledge proofs are zkSNARK, zkSTARKs, BulletProofs, or Groth16.

6. The data decentralized storage and disaster recovery processing method as described in claim 1, characterized in that: The malicious node is a data node that does not store the original data correctly, or a redundant node that does not calculate the data correctly, or an inactive node that does not provide verification results or cannot respond to the BFT consensus in a timely manner.

7. The data decentralized storage and disaster recovery processing method as described in claim 6, characterized in that: A node is removed when it fails to provide a verification result or respond to BFT consensus in a timely manner a preset number of times.

8. A computer device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the data decentralized storage and disaster recovery processing method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data decentralized storage and disaster recovery processing method as described in any one of claims 1-7.

Images