A data redundancy method, device, apparatus, storage medium and program product
Patent Information
- Application Number
- CN202611508282.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-10-30
AI Technical Summary
[0003]本申请提供了一种数据冗余方法、装置、设备、存储介质及程序产品，以至少解决相关技术不能适用于任意物理节点数的集群的问题
[0009]有益效果是：本申请提供的数据冗余方法，为每个物理节点创建虚拟节点，并将虚拟节点分组后，在组内确定虚拟节点对应的复制目标虚拟节点与复制源虚拟节点，进而对虚拟节点与对应的复制目标虚拟节点与复制源虚拟节点彼此进行数据的备份。这样通过虚拟节点将物理节点数与对称平衡不完全块设计组成员数解耦，适用于任意物理节点数的集群，并且支持增加或移除物理节点。
Smart Images
Figure CN123011610A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to a data redundancy method, apparatus, device, storage medium, and program product. Background Technology
[0002] Nodes in a distributed storage system may encounter various types of failures, leading to data loss. To ensure data availability, fault tolerance techniques are typically employed, with data redundancy being a primary method. Data redundancy includes data replication and erasure coding. For erasure coding design, related technologies employ striped data placement schemes. However, these technologies have significant limitations and are not applicable to clusters with an arbitrary number of physical nodes. Therefore, addressing these technical shortcomings has become a pressing technical problem for those skilled in the art. Summary of the Invention
[0003] This application provides a data redundancy method, apparatus, device, storage medium, and program product to at least solve the problem that related technologies cannot be applied to clusters with an arbitrary number of physical nodes.
[0004] This application provides a data redundancy method, including: Create multiple virtual nodes on each physical node; The virtual nodes are divided into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node. Within the symmetric balanced incomplete block design group, determine the target virtual node and the source virtual node corresponding to the virtual node; Select a target virtual node from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node; The virtual node receives and caches data backups from the corresponding replication source virtual node, and stores a verification block generated based on the received data backups on the virtual node.
[0005] This application also provides a data redundancy device, including: Create a module to create multiple virtual nodes on each physical node; The partitioning module is used to divide virtual nodes into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node. The determination module is used to determine the target virtual node and the source virtual node for replication corresponding to the virtual node within the symmetric balance incomplete block design group; The sending module is used to select a target virtual node for replication from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node for replication; A receiving module is configured to receive and cache data backups from the corresponding replication source virtual node in the virtual node, and to store a verification block generated based on the received data backups in the virtual node.
[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing any of the above-described data redundancy methods when executing the computer program.
[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data redundancy methods.
[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data redundancy methods.
[0009] The beneficial effects are as follows: The data redundancy method provided in this application creates a virtual node for each physical node, groups the virtual nodes, and determines the corresponding target and source virtual nodes for each virtual node within the group. This allows for data backup between the virtual nodes and their corresponding target and source virtual nodes. By using virtual nodes, the number of physical nodes is decoupled from the number of members in the symmetric balanced incomplete block design group, making it suitable for clusters with any number of physical nodes and supporting the addition or removal of physical nodes.
[0010] The data redundancy device, equipment, storage medium, and program products provided in this application also have the aforementioned technical effects. Attached Figure Description
[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 A flowchart illustrating a data redundancy method provided in an embodiment of this application; Figure 2This application provides a schematic diagram of a data backup and verification block generation method. Figure 3 A schematic diagram of virtual node grouping provided in an embodiment of this application; Figure 4 This application provides a schematic diagram of data migration when adding physical nodes, as illustrated in an embodiment of the present application. Figure 5 This application provides a schematic diagram of data migration when removing a physical node. Figure 6 This is a schematic diagram of a data redundancy device provided in an embodiment of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0016] The embodiments of this application provide a data redundancy method. The method is described in detail below in conjunction with the execution flow of the method.
[0017] refer to Figure 1 As shown, an embodiment of this application provides a data redundancy method including: S101: Create multiple virtual nodes on each physical node.
[0018] Creating multiple virtual nodes on each physical node includes: Create v virtual nodes on each of the n physical nodes; n > v.
[0019] Initialization Create on each physical node A virtual node. > .
[0020] In some embodiments, the method further includes: using a virtual consistent hashing algorithm to distribute all the created virtual nodes on a hash ring.
[0021] Using a virtual consistent hashing algorithm, all Virtual nodes are distributed across the hash ring for data storage and retrieval. Distributing virtual nodes across the hash ring allows data to be more evenly distributed across physical nodes, avoiding hotspot issues. When a node fails, the load is smoothly migrated to multiple other nodes, preventing the concentrated pressure on a single node. When adding or removing nodes, only a small amount of data needs to be migrated, thus localizing the impact.
[0022] remember The physical nodes are numbered as follows For each physical node Create on it There are several logical virtual nodes, denoted as follows: The entire cluster has a total of There are 10 virtual nodes, each associated with a physical node. All data access requests on a virtual node, including data read / write, backup transmission, checksum calculation and storage, are redirected to the physical node associated with that virtual node, where the actual operation is performed.
[0023] S102: Divide the virtual nodes into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node.
[0024] Create v virtual nodes on n physical nodes; n>v; Accordingly, the virtual nodes are divided into n symmetric balanced incomplete block design groups, each symmetric balanced incomplete block design group includes v virtual nodes; the v virtual nodes in each symmetric balanced incomplete block design group come from v different physical nodes.
[0025] Will The virtual nodes are divided into There are 10 SBIBD (Symmetric Balanced Incomplete Block Designs) groups, each containing 10 SBIBD groups. There are 10 virtual nodes. To ensure the cluster's fault tolerance, the mutual exclusion principle must be satisfied: any two virtual nodes within a symmetric balanced incomplete block design group do not belong to the same physical node; that is, any virtual node within a symmetric balanced incomplete block design group... The virtual nodes come from A different physical node. This ensures that the mutual exclusion principle can be satisfied.
[0026] In some embodiments, dividing virtual nodes into multiple symmetric balanced incomplete block design groups includes: The number of the symmetric balanced incomplete block design group to which the virtual node belongs is determined based on the number of the virtual node, the total number of physical nodes, and the number of virtual nodes created on a physical node. The virtual nodes are assigned to the symmetric balanced incomplete block design group corresponding to the number.
[0027] In some embodiments, determining the number of the symmetric balanced incomplete block design group to which the virtual node belongs, based on the virtual node's number, the total number of physical nodes, and the number of virtual nodes created on a single physical node, includes: Multiply the sequence number of the virtual node in its physical node parent by the total number of physical nodes; the number of the virtual node includes the number of the physical node to which the virtual node belongs and the sequence number of the virtual node in its physical node parent. Add the resulting product to the number of the physical node to which the virtual node belongs; Divide the sum by the number of virtual nodes contained in a physical node; Round the obtained quotient down to get the symmetric balance incomplete block design group number to which the virtual node belongs.
[0028] serial number Represents physical nodes The There are virtual nodes. For any virtual node The method for determining the numbering of the symmetric equilibrium incomplete block design group to which it belongs is as follows: .
[0029] in, This indicates the design group number for the symmetrically balanced incomplete block design. Indicates not greater than The largest integer. Through the method for determining the numbering of the symmetric balanced incomplete block design group to which the virtual node belongs, all... Each virtual node was assigned a number. of In a group of symmetrically balanced incomplete block designs.
[0030] By dividing virtual nodes into multiple symmetric balanced incomplete block design groups as provided in this embodiment, it can be ensured that all symmetric balanced incomplete block design groups satisfy the mutual exclusion principle.
[0031] In some embodiments, it also includes: Create a member table for the symmetric balanced incomplete block design group and store the member table in the global metadata.
[0032] To ensure that the mutual exclusion principle is always satisfied during cluster operation, this embodiment creates a member table for each symmetric balanced incomplete block design group, recording the IDs of all virtual nodes within the symmetric balanced incomplete block design group. .serial number Represents physical nodes The virtual nodes ( , All can be counted starting from 0). The member tables of all symmetric balanced incomplete block design groups are stored in the global metadata. When a symmetric balanced incomplete block design group is created (symmetric balanced incomplete block design group creation occurs during cluster initialization), the member table of that symmetric balanced incomplete block design group does not contain two virtual node numbers. , making When a virtual node is migrated to a physical node (Virtual node migration occurs when a physical node is added or removed), and the virtual node number does not exist in the member table of the symmetric balanced incomplete block design group to which the virtual node belongs. , making .
[0033] S103: Within the symmetric balanced incomplete block design group, determine the target virtual node and the source virtual node corresponding to the virtual node.
[0034] The target virtual node is used to back up data from other virtual nodes, while the source virtual node is the virtual node whose data is to be backed up by other virtual nodes. A virtual node, its corresponding target virtual node, and its corresponding source virtual node belong to the same symmetric balanced incomplete block design group.
[0035] In some embodiments, determining the target virtual node and source virtual node for a virtual node within the symmetric balanced incomplete block design group includes: Based on the index of the virtual node, a virtual node is selected from the symmetric balanced incomplete block design matrix to obtain the set of replication sources of the virtual node; Select virtual nodes corresponding to multiple elements in the set of replication sources as the virtual nodes of the replication sources; Based on the index of the virtual node, a virtual node is selected from the symmetric balanced incomplete block design matrix to obtain the set of replication targets for the virtual node; Select virtual nodes corresponding to multiple elements in the set of replication targets as the replication target virtual nodes.
[0036] In some embodiments, each row and column of the symmetric balanced incomplete block design matrix has a preset number of 1s, and the remaining positions are 0s. For any two rows, there is a column such that the corresponding positions of these two rows are all 1s, and the values of the elements on the diagonal of the symmetric balanced incomplete block design matrix are all 0s.
[0037] In some embodiments, virtual nodes are selected from the symmetric balanced incomplete block design matrix according to their indices to obtain the set of replication sources for the virtual nodes, including: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the column corresponding to the index of the virtual node to obtain the set of replication sources of the virtual node.
[0038] In some embodiments, virtual nodes are selected from the symmetric balanced incomplete block design matrix according to their indices to obtain the set of replication targets for the virtual nodes, including: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the row corresponding to the virtual node's index to obtain the set of replication targets for the virtual node.
[0039] The embodiments of this application are based on The SBIBD matrix defines the target virtual node and source virtual node for each virtual node. The possible value is 1.
[0040] One parameter is The SBIBD matrix is a 0 / 1 matrix. Each row and each column has (That is, the preset number mentioned above) 1s, and the rest are 0s. For any two rows, there is one column such that the corresponding positions in these two rows are all 1s. For example, when hour, An SBIBD matrix is shown below: .
[0041] The diagonal elements of the SBIBD matrix are all 0 because nodes cannot copy data to themselves.
[0042] Note the design group of symmetrically balanced incomplete blocks. The virtual node is numbered as follows , -SBIBD matrix is Among them, the matrix can be made The rows and columns are counted starting from 0. .
[0043] For any virtual node , is called a set For virtual nodes The set of copy sources, in the set of copy sources The virtual node corresponding to each element is a virtual node. The source virtual node for replication.
[0044] For any virtual node , is called a set For virtual nodes The set of targets to be copied, in the set of targets to be copied The virtual node corresponding to each element is a virtual node. The target virtual node for replication.
[0045] S104: Select a target virtual node from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node.
[0046] S105: Receive and cache data backups from the corresponding replication source virtual node in the virtual node, and save a check block generated based on the received data backups in the virtual node.
[0047] In some embodiments, selecting a target virtual node from the target virtual nodes corresponding to the virtual node and sending the data backup on the virtual node to the selected target virtual node includes: Select p+1 target virtual nodes from the virtual nodes corresponding to the virtual nodes, and send the data backup on the virtual nodes to the selected p+1 target virtual nodes, where p is the fault tolerance threshold.
[0048] The embodiments of this application are based on parameters. Complete the data copying and checksum generation on each virtual node.
[0049] parameter This represents the fault tolerance threshold, which is the threshold value of the virtual node containing each data block. Each replica target virtual node sends a backup of the data block, used to tolerate... One node failed. Each replication target virtual node replicates data from the virtual node where the data block resides. Arbitrarily select from the target virtual nodes for replication. Parameters The impact on storage overhead is that each virtual node derives from its... Each virtual node from the replication source receives data backups and temporarily caches them. When it receives data from all... When backing up the data blocks of a replica source virtual node, this Each backup is transformed into a single checksum block through an XOR operation. The processes of sending data block backups and generating checksum blocks are asynchronous. For example, refer to... Figure 2 As shown, the data on a certain virtual node One backup sent to A replica target virtual node. A certain virtual node receives data from... Data backups from each source virtual node are temporarily stored in a buffer space. When data is received from all... When backing up the data blocks of a replica source virtual node, this Each backup is converted into a single checksum block using an XOR operation and saved; the original backup can then be deleted. One data backup.
[0050] In some embodiments, it also includes: Select physical nodes from the physical nodes and then select virtual nodes from the selected physical nodes; Check whether the selected virtual nodes and the virtual nodes on the newly added physical nodes do not belong to the same symmetric balanced incomplete block design group; If so, migrate the data, data backup, and verification blocks on the selected virtual node to the newly added physical node, and modify the number of the selected virtual node.
[0051] In some embodiments, it also includes: Modify the member table of the symmetric balanced incomplete block design group to which the selected virtual node belongs.
[0052] The embodiments of this application aim to migrate virtual nodes (including data, data backups, and check blocks on the virtual nodes) when physical nodes are added to the cluster, while maintaining the fault tolerance capability of the cluster to tolerate the failure of p physical nodes.
[0053] When a new physical node is added to the cluster, one virtual node (including its data, temporary backups (data backups from other virtual nodes), and checksum blocks) can be migrated from several previously heavily loaded physical nodes to the new physical node. This requires that all virtual nodes on the new physical node after migration belong to different symmetric-balanced incomplete block design groups. Data on other virtual nodes that are not migrated will not be affected, thus achieving efficient data migration.
[0054] Record the newly added physical node number as The process for migrating virtual nodes is as follows: Select a physical node from which you intend to migrate some of the data. (Typically, this could be a physical node with a high load), assuming it has corresponding physical nodes. There are 1 virtual node, numbered as follows: Assuming a new physical node... There is already A virtual node (which may be a virtual node that was previously migrated from other physical nodes, or...) Preparing to set up virtual nodes. Remapping to the newly added physical nodes First, check the virtual nodes. Is it related to physical nodes? All of the above None of the virtual nodes belong to the same symmetric balanced incomplete block design group. If the check passes ( (Automatic access at any time), then the virtual node will be... Migrate to physical node Above, and renumbered as Simultaneously modify the member table of the symmetric balanced incomplete block design group containing the virtual node in the metadata. If the check fails, abandon the process from the physical node. Migrate virtual nodes.
[0055] For other physical nodes that are ready to migrate some data (usually physical nodes with high load), the above process is executed in sequence.
[0056] This application's embodiments support adding physical nodes. The above migration process ensures that each symmetric balanced incomplete block design group still maintains the mutual exclusion principle after adding physical nodes, thereby maintaining system tolerance. The ability to handle physical node failures remains unchanged.
[0057] In some embodiments, it also includes: Find the target physical node from the list of physical nodes that are not removed; the target physical node satisfies the condition that the virtual nodes to be migrated on the physical node to be removed and the virtual nodes on the target physical node do not belong to the same symmetric balanced incomplete block design group. Migrate the data, data backups, and verification blocks on the virtual node to be migrated to the target physical node.
[0058] In some embodiments, it also includes: Modify the member table of the symmetric balanced incomplete block design group where the virtual node to be migrated is located.
[0059] The embodiments of this application aim to migrate data to suitable other physical nodes when a physical node is removed from the cluster, while maintaining system tolerance. The ability to handle physical node failures remains unchanged. When the cluster prepares to remove a physical node, a suitable physical node to be migrated is first selected for each virtual node on that physical node. This virtual node must satisfy the condition that none of the virtual nodes on the physical node to be migrated belong to the same symmetric balanced incomplete block design group as any of the virtual nodes on the physical node to be migrated. Once all virtual nodes on the physical node to be removed (including data on that virtual node, temporary backups (i.e., data backups from other virtual nodes), and checksum blocks) have been migrated to the corresponding physical node, the physical node to be removed can be removed.
[0060] When a physical node needs to be removed (provided that the number of remaining physical nodes after removal is still not less than...), The virtual nodes (including data, temporary backups, and checksums) on these nodes are migrated to other suitable physical nodes. Data on other virtual nodes that are not migrated remains unaffected, thus achieving efficient data migration.
[0061] The physical node to be removed is numbered as follows: Currently, it has There are 1 virtual node, numbered as follows: The physical node to be removed is numbered... Each virtual node Perform the following migration process: Search the remaining physical nodes in the cluster sequentially. For physical nodes... Assuming it currently has A virtual node, check the virtual node. Is it related to physical nodes? All of the above None of the virtual nodes belong to the same symmetric balanced incomplete block design group. If the check fails, search for the next physical node until a physical node that passes the check is found (still denoted as...). Since, excluding the physical nodes to be removed, the number of remaining physical nodes is not less than the number of members in the symmetric balanced incomplete block design team. Therefore, there must exist physical nodes that can pass the inspection. Virtual nodes... Migrate to physical node Above, and renumbered as Simultaneously modify the member table of the symmetric balanced incomplete block design group to which the virtual node belongs in the metadata.
[0062] For physical nodes to be removed For the remaining virtual nodes, perform the above process sequentially until the physical node to be removed is reached. If all virtual nodes have been migrated, the physical node to be removed can be safely removed.
[0063] The embodiments of this application support the removal of physical nodes, and the above migration process ensures that each symmetric unbalanced incomplete block design group still maintains the mutual exclusion principle after the removal of physical nodes, thereby maintaining system tolerance. The ability to handle physical node failures remains unchanged.
[0064] In some embodiments, it also includes: Locate the symmetric unbalanced incomplete block design group where the faulty virtual node is located on the faulty physical node; Based on the data on the fault-free virtual nodes in the symmetric balanced incomplete block design group where the faulty virtual node is located, the data on the faulty virtual node is recovered from the check block.
[0065] This application's embodiments aim to achieve data recovery in the event of physical node failure. For each symmetric balanced incomplete block design group, when the number of its failed virtual nodes does not exceed... In such cases, faulty data can be repaired through iterative recovery from checksum blocks and live data blocks. When the cluster has no more than When a physical node fails, the number of failed virtual nodes in each symmetric incomplete block design group does not exceed [number missing]. Therefore, when the cluster has no more than When a physical node fails, the lost data is recovered within the symmetric unbalanced incomplete block design group of each failed physical node, thus completing the data recovery on all failed physical nodes.
[0066] The following is For example, a specific implementation is described: The distributed storage system has 4 physical nodes and uses parameters. Perform barless erasure coding, parameters ,satisfy Create 3 virtual nodes on each physical node, dividing the total 12 virtual nodes into 4 symmetric balanced incomplete block design groups. Within each group, based on parameters... and The SBIBD matrix determines the set of replication sources and replication targets for each virtual node.
[0067] like Figure 3As shown, three virtual nodes are created on physical node Node0, denoted as 0-0, 0-1, and 0-2. Three virtual nodes are created on physical node Node1, denoted as 1-0, 1-1, and 1-2. Three virtual nodes are created on physical node Node2, denoted as 2-0, 2-1, and 2-2. Three virtual nodes are created on physical node Node3, denoted as 3-0, 3-1, and 3-2. Virtual nodes 0-1, 1-1, and 3-0 are grouped into a symmetric balanced incomplete block design group. For virtual node 0-1, virtual nodes 1-1 and 3-0 are its corresponding replication target virtual nodes, and virtual nodes 1-1 and 3-0 are its corresponding replication source virtual nodes. Data backups of virtual node 0-1 are sent to virtual nodes 1-1 and 3-0, and virtual node 0-1 receives data backups sent by virtual nodes 1-1 and 3-0, and generates a check block based on the received data backups.
[0068] When the system needs to add a physical node Node4, it attempts to migrate one virtual node from Node0 to Node3 to Node4.
[0069] refer to Figure 4 As shown, the current number of virtual nodes on Node 4 is 0. For virtual node 0-2 on Node 0, there are no virtual nodes on Node 4 belonging to the same symmetric balanced incomplete block design group as virtual node 0-2. Therefore, virtual node 0-2 is redirected to Node 4, the data, temporary backups, and check blocks on virtual node 0-2 are migrated, virtual node 0-2 is renamed to 4-0, and the member table in the global metadata is modified.
[0070] Currently, there is one virtual node on Node4. Virtual nodes 1-2 on Node1 and 4-0 on Node4 do not belong to the same symmetric balanced incomplete block design group as virtual nodes 1-2. Therefore, virtual node 1-2 is redirected to Node4, its data, temporary backups, and check blocks are migrated, and virtual node 1-2 is renamed to 4-1. The member table in the global metadata is also modified.
[0071] The current number of virtual nodes on Node4 is 2. For virtual node 2-2 on Node2, virtual node 4-1 on Node4 and virtual node 2-2 belong to the same symmetric balanced incomplete block design group, so we abandon the migration of data from Node2. Similarly, for virtual node 3-2 on Node3, virtual node 4-1 on Node4 and 3-2 belong to the same symmetric balanced incomplete block design group, so we abandon the migration of data from Node3.
[0072] When the system needs to remove a physical node Node3, all virtual nodes on Node3 will be migrated to the other three physical nodes.
[0073] refer to Figure 5 As shown, for virtual node 3-0, checking other physical nodes sequentially reveals that virtual node 0-1 on Node0 belongs to the same symmetric balanced incomplete block design group as virtual node 3-0, and virtual node 1-1 on Node1 also belongs to the same symmetric balanced incomplete block design group as virtual node 3-0. Node2 has no virtual nodes belonging to the first group containing virtual node 3-0. Therefore, virtual node 3-0 is redirected to Node2, its data, temporary backups, and checksum blocks are migrated, virtual node 3-0 is renamed to 2-3, and the member table in the global metadata is modified.
[0074] For virtual node 3-1, the other physical nodes are checked sequentially. It is found that virtual node 0-2 on Node0 belongs to the same symmetric balanced incomplete block design group as virtual node 3-1, while there is no virtual node on Node1 belonging to the second group to which virtual node 3-1 belongs. Therefore, virtual node 3-1 is redirected to Node1, the data, temporary backups, and check blocks on virtual node 3-1 are migrated, virtual node 3-1 is renamed to 1-3, and the member table in the global metadata is modified.
[0075] For virtual node 3-2, the other physical nodes are checked in turn. It is found that there is no virtual node in the 3rd group that belongs to virtual node 3-2 on Node0. Therefore, virtual node 3-2 is redirected to Node0, the data, temporary backup and check block on virtual node 3-2 are migrated, virtual node 3-2 is renamed to 0-3, and the member table in the global metadata is modified.
[0076] After migrating all virtual nodes on Node3, remove the physical node Node3.
[0077] Figure 3 The system shown can tolerate a maximum of one physical node failure.
[0078] If Node0 fails, data on virtual nodes 0-0, 0-1, and 0-2 needs to be recovered separately. The failed virtual node 0-0 belongs to group 0, and for any one of the failed data nodes... There exists a fault-free check value corresponding to virtual node 1-0. And the corresponding fault-free data on virtual node 2-0 , making Therefore, the faulty data on the faulty virtual node 0-0 can be recovered using the data and checksums from the other fault-free virtual nodes in group 0. Similarly, virtual nodes 0-1 and 0-2 can be recovered using the other virtual nodes in groups 1 and 2, respectively. Thus, all faulty data of the faulty physical node Node0 can be recovered.
[0079] For systems with a larger number of nodes and a greater tolerance for failures, the faulty data in a faulty virtual node may not be directly recoverable by finding a set of fault-free checksums and data. Instead, iterative calculations are required for recovery. The iterative calculation process will not be described in detail here.
[0080] In summary, the data redundancy method provided in this application creates a virtual node for each physical node, groups the virtual nodes, and determines the corresponding target and source virtual nodes for each virtual node within the group. This allows for data backup between the virtual node and its corresponding target and source virtual nodes. By using virtual nodes, the number of physical nodes is decoupled from the number of members in the symmetric balanced incomplete block design group, making it suitable for clusters with any number of physical nodes and supporting the addition or removal of physical nodes. This method can be applied to scenarios such as fault-tolerant design in distributed memory storage systems.
[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0082] Embodiments of this application also provide a data redundancy device, referencing Figure 6 As shown, the device includes: Create module 10, which is used to create multiple virtual nodes on each physical node; The partitioning module 20 is used to divide the virtual nodes into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node. Module 30 is used to determine the target virtual node and source virtual node for replication of virtual nodes within the symmetric balanced incomplete block design group. The sending module 40 is used to select a target virtual node for replication from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node for replication; The receiving module 50 is configured to receive and cache data backups from the corresponding replication source virtual node in the virtual node, and to store a verification block generated based on the received data backups in the virtual node.
[0083] Based on the above embodiments, as a specific implementation method, it further includes: The selection module is used to select physical nodes from each physical node and virtual nodes from the selected physical nodes. The inspection module is used to check whether the selected virtual nodes and the virtual nodes on the newly added physical nodes do not belong to the same symmetric balanced incomplete block design group. The first migration module is used to migrate the data, data backup, and verification blocks on the selected virtual node to the newly added physical node if the condition is met, and to modify the number of the selected virtual node.
[0084] Based on the above embodiments, as a specific implementation method, it further includes: The first search module is used to find the target physical node from the list of physical nodes that are not removed; the target physical node satisfies the condition that the virtual nodes to be migrated on the physical node to be removed and the virtual nodes on the target physical node do not belong to the same symmetric balanced incomplete block design group. The second migration module is used to migrate the data, data backup, and verification blocks on the virtual node to be migrated to the target physical node.
[0085] Based on the above embodiments, as a specific implementation method, the partitioning module 20 includes: The numbering determination unit is used to determine the number of the symmetric balanced incomplete block design group to which the virtual node belongs, based on the number of the virtual node, the total number of physical nodes, and the number of virtual nodes created on a physical node. A partitioning unit is used to partition the virtual node into the symmetric balanced incomplete block design group corresponding to the number.
[0086] Based on the above embodiments, as a specific implementation method, the numbering determination unit includes: The first calculation subunit is used to multiply the sequence number of the virtual node in its physical node by the total number of physical nodes; the number of the virtual node includes the number of the physical node to which the virtual node belongs and the sequence number of the virtual node in its physical node. The second calculation subunit is used to add the obtained product to the number of the physical node to which the virtual node belongs; The third calculation subunit is used to divide the resulting sum by the number of virtual nodes contained in a physical node; The fourth calculation subunit is used to round down the obtained quotient to obtain the design group number of the symmetric balanced incomplete block to which the virtual node belongs.
[0087] Based on the above embodiments, as a specific implementation method, the determining module 30 includes: The first selection unit is used to select virtual nodes from the symmetric balanced incomplete block design matrix according to the index of the virtual nodes, and obtain the set of replication sources of the virtual nodes. The second selection unit is used to select virtual nodes corresponding to multiple elements in the copy source set as the copy source virtual nodes; The third selection unit is used to select virtual nodes from the symmetric balanced incomplete block design matrix according to the serial number of the virtual nodes, so as to obtain the set of replication targets of the virtual nodes. The fourth selection unit is used to select virtual nodes corresponding to multiple elements in the copy target set as the copy target virtual nodes.
[0088] Based on the above embodiments, as a specific implementation method, each row and column of the symmetric balanced incomplete block design matrix has a preset number of 1s, and the remaining positions are 0s. For any two rows, there is a column such that the corresponding positions of these two rows are all 1s, and the values of the elements on the diagonal of the symmetric balanced incomplete block design matrix are all 0s.
[0089] Based on the above embodiments, as a specific implementation method, the first selection unit is used for: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the column corresponding to the index of the virtual node to obtain the set of replication sources of the virtual node.
[0090] Based on the above embodiments, as a specific implementation method, the third selection unit is used for: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the row corresponding to the virtual node's index to obtain the set of replication targets for the virtual node.
[0091] Based on the above embodiments, as a specific implementation method, it further includes: The member table creation module is used to create member tables for symmetric balanced incomplete block design groups and store the member tables in the global metadata.
[0092] Based on the above embodiments, as a specific implementation method, it further includes: The first modification module is used to modify the member table of the symmetric balanced incomplete block design group to which the selected virtual node belongs.
[0093] Based on the above embodiments, as a specific implementation method, it further includes: The second modification module is used to modify the member table of the symmetric balanced incomplete block design group where the virtual node to be migrated is located.
[0094] Based on the above embodiments, as a specific implementation method, it further includes: The second search module is used to find the symmetric unbalanced incomplete block design group where the faulty virtual node is located on the faulty physical node. The data recovery module is used to recover the data on the faulty virtual node based on the data on the fault-free virtual node and the check block within the symmetric unbalanced incomplete block design group where the faulty virtual node is located.
[0095] Based on the above embodiments, as a specific implementation method, module 10 is used for: Create v virtual nodes on n physical nodes; n>v; Accordingly, module 20 is used for: The virtual nodes are divided into n symmetric balanced incomplete block design groups, each of which includes v virtual nodes; the v virtual nodes in each symmetric balanced incomplete block design group come from v different physical nodes.
[0096] Based on the above embodiments, as a specific implementation method, it further includes: The distributed module is used to distribute all created virtual nodes across a hash ring using a virtual consistent hashing algorithm.
[0097] Based on the above embodiments, as one specific implementation, the sending module 40 is used for: Select p+1 target virtual nodes from the virtual nodes corresponding to the virtual nodes, and send the data backup on the virtual nodes to the selected p+1 target virtual nodes, where p is the fault tolerance threshold.
[0098] The data redundancy device provided in this application creates a virtual node for each physical node, groups the virtual nodes, and determines the corresponding target virtual node and source virtual node for each virtual node within the group. This allows for data backup between the virtual node and its corresponding target and source virtual nodes. By using virtual nodes, the number of physical nodes is decoupled from the number of members in the symmetric balanced incomplete block design group, making it suitable for clusters with any number of physical nodes and supporting the addition or removal of physical nodes.
[0099] For a description of the features in the embodiment corresponding to the data redundancy device, please refer to the relevant description in the embodiment corresponding to the data redundancy method, which will not be repeated here.
[0100] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the following steps: Multiple virtual nodes are created on each physical node; the virtual nodes are divided into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node; within the symmetric balanced incomplete block design group, the target virtual node and the source virtual node corresponding to the virtual node are determined; a target virtual node is selected from the target virtual nodes corresponding to the virtual node, and the data backup on the virtual node is sent to the selected target virtual node; the data backup from the source virtual node corresponding to the virtual node is received and cached on the virtual node, and a check block generated based on the received data backup is stored on the virtual node.
[0101] The electronic device provided in this application creates a virtual node for each physical node, groups the virtual nodes, and determines the corresponding target virtual node and source virtual node for replication within each group. This allows for data backup between the virtual node and its corresponding target and source virtual nodes. By using virtual nodes, the number of physical nodes is decoupled from the number of members in the symmetric balanced incomplete block design group, making it suitable for clusters with any number of physical nodes and supporting the addition or removal of physical nodes.
[0102] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the following steps when run: Multiple virtual nodes are created on each physical node; the virtual nodes are divided into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node; within the symmetric balanced incomplete block design group, the target virtual node and the source virtual node corresponding to the virtual node are determined; a target virtual node is selected from the target virtual nodes corresponding to the virtual node, and the data backup on the virtual node is sent to the selected target virtual node; the data backup from the source virtual node corresponding to the virtual node is received and cached on the virtual node, and a check block generated based on the received data backup is stored on the virtual node.
[0103] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0104] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, performs the following steps: Multiple virtual nodes are created on each physical node; the virtual nodes are divided into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node; within the symmetric balanced incomplete block design group, the target virtual node and the source virtual node corresponding to the virtual node are determined; a target virtual node is selected from the target virtual nodes corresponding to the virtual node, and the data backup on the virtual node is sent to the selected target virtual node; the data backup from the source virtual node corresponding to the virtual node is received and cached on the virtual node, and a check block generated based on the received data backup is stored on the virtual node.
[0105] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps: Multiple virtual nodes are created on each physical node; the virtual nodes are divided into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node; within the symmetric balanced incomplete block design group, the target virtual node and the source virtual node corresponding to the virtual node are determined; a target virtual node is selected from the target virtual nodes corresponding to the virtual node, and the data backup on the virtual node is sent to the selected target virtual node; the data backup from the source virtual node corresponding to the virtual node is received and cached on the virtual node, and a check block generated based on the received data backup is stored on the virtual node.
[0106] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.
[0107] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0108] The foregoing has provided a detailed description of a data redundancy method, apparatus, device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.
Claims
1. A data redundancy method, characterized in that, include: Create multiple virtual nodes on each physical node; The virtual nodes are divided into multiple symmetric balanced incomplete block design groups; A symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node. Within the symmetric balanced incomplete block design group, determine the target virtual node and the source virtual node for the replication of the virtual node; Select a target virtual node from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node; The virtual node receives and caches data backups from the corresponding replication source virtual node, and stores a verification block generated based on the received data backups on the virtual node.
2. The data redundancy method according to claim 1, characterized in that, Also includes: Select physical nodes from the physical nodes and then select virtual nodes from the selected physical nodes; Check whether the selected virtual nodes and the virtual nodes on the newly added physical nodes do not belong to the same symmetric balanced incomplete block design group; If so, migrate the data, data backup, and verification blocks on the selected virtual node to the newly added physical node, and modify the number of the selected virtual node.
3. The data redundancy method according to claim 1, characterized in that, Also includes: Find the target physical node from the list of physical nodes that are not removed; the target physical node satisfies the condition that the virtual nodes to be migrated on the physical node to be removed and the virtual nodes on the target physical node do not belong to the same symmetric balanced incomplete block design group. The data, data backup, and verification block on the virtual node to be migrated are migrated to the target physical node.
4. The data redundancy method according to claim 1, characterized in that, The virtual nodes are divided into multiple symmetric balanced incomplete block design groups, including: The number of the symmetric balanced incomplete block design group to which the virtual node belongs is determined based on the number of the virtual node, the total number of physical nodes, and the number of virtual nodes created on a physical node. The virtual nodes are assigned to the symmetric balanced incomplete block design group corresponding to the determined number.
5. The data redundancy method according to claim 4, characterized in that, The number of the symmetric balanced incomplete block design group to which the virtual node belongs is determined based on the virtual node's number, the total number of physical nodes, and the number of virtual nodes created from a single physical node. Multiply the sequence number of the virtual node in its physical node parent by the total number of physical nodes; the number of the virtual node includes the number of the physical node to which the virtual node belongs and the sequence number of the virtual node in its physical node parent. Add the resulting product to the number of the physical node to which the virtual node belongs; Divide the sum by the number of virtual nodes contained in a physical node; Round the obtained quotient down to get the symmetric balance incomplete block design group number to which the virtual node belongs.
6. The data redundancy method according to claim 1, characterized in that, Determining the target virtual node and source virtual node for a virtual node within the symmetric balanced incomplete block design group includes: Based on the index of the virtual node, a virtual node is selected from the symmetric balanced incomplete block design matrix to obtain the set of replication sources of the virtual node; Select virtual nodes corresponding to multiple elements in the set of replication sources as the virtual nodes of the replication sources; Based on the index of the virtual node, a virtual node is selected from the symmetric balanced incomplete block design matrix to obtain the set of replication targets for the virtual node; Select virtual nodes corresponding to multiple elements in the set of replication targets as the replication target virtual nodes.
7. The data redundancy method according to claim 6, characterized in that, The symmetric balanced incomplete block design matrix has a preset number of 1s in each row and column, and the remaining positions are 0. For any two rows, there is a column such that the corresponding positions of these two rows are all 1s, and the values of the elements on the diagonal of the symmetric balanced incomplete block design matrix are all 0s.
8. The data redundancy method according to claim 7, characterized in that, Virtual nodes are selected from the symmetric balanced incomplete block design matrix based on their indices, resulting in a set of replication sources for the virtual nodes, including: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the column corresponding to the index of the virtual node to obtain the set of replication sources of the virtual node.
9. The data redundancy method according to claim 7, characterized in that, Virtual nodes are selected from the symmetric balanced incomplete block design matrix based on their indices, resulting in a set of replication targets for the virtual nodes, including: From the symmetric balanced incomplete block design matrix, select the virtual node corresponding to the element with a value of 1 in the row corresponding to the virtual node's index to obtain the set of replication targets for the virtual node.
10. The data redundancy method according to claim 1, characterized in that, Also includes: Create a member table for the symmetric balanced incomplete block design group and store the member table in the global metadata.
11. The data redundancy method according to claim 2, characterized in that, Also includes: Modify the member table of the symmetric balanced incomplete block design group to which the selected virtual node belongs.
12. The data redundancy method according to claim 3, characterized in that, Also includes: Modify the member table of the symmetric balanced incomplete block design group where the virtual node to be migrated is located.
13. The data redundancy method according to claim 1, characterized in that, Also includes: Locate the symmetric unbalanced incomplete block design group where the faulty virtual node is located on the faulty physical node; Based on the data on the fault-free virtual nodes in the symmetric balanced incomplete block design group where the faulty virtual node is located, the data on the faulty virtual node is recovered from the check block.
14. The data redundancy method according to claim 1, characterized in that, Creating multiple virtual nodes on each physical node includes: Create v virtual nodes on n physical nodes; n>v; Accordingly, the virtual nodes are divided into multiple symmetric balanced incomplete block design groups, including: The virtual nodes are divided into n symmetric balanced incomplete block design groups, each of which includes v virtual nodes; the v virtual nodes in each symmetric balanced incomplete block design group come from v different physical nodes.
15. The data redundancy method according to claim 1, characterized in that, Also includes: A virtual consistent hashing algorithm is used to distribute all the created virtual nodes across a hash ring.
16. The data redundancy method according to claim 1, characterized in that, Selecting a target virtual node from the virtual nodes corresponding to the virtual node and sending the data backup on the virtual node to the selected target virtual node includes: Select p+1 target virtual nodes from the virtual nodes corresponding to the virtual nodes, and send the data backup on the virtual nodes to the selected p+1 target virtual nodes, where p is the fault tolerance threshold.
17. A data redundancy device, characterized in that, include: Create a module to create multiple virtual nodes on each physical node; The partitioning module is used to divide virtual nodes into multiple symmetric balanced incomplete block design groups; a symmetric balanced incomplete block design group refers to a combination of virtual nodes in which any two virtual nodes do not belong to the same physical node. The determination module is used to determine the target virtual node and the source virtual node for replication corresponding to the virtual node within the symmetric balance incomplete block design group. The sending module is used to select a target virtual node for replication from the target virtual nodes corresponding to the virtual node, and send the data backup on the virtual node to the selected target virtual node for replication; A receiving module is configured to receive and cache data backups from the corresponding replication source virtual node in the virtual node, and to store a verification block generated based on the received data backups in the virtual node.
18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for implementing the steps of the data redundancy method as described in any one of claims 1 to 16 when executing the computer program.
19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data redundancy method as described in any one of claims 1 to 16.
20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data redundancy method as described in any one of claims 1 to 16.