Ignite-based decentralized rack-aware method, system, electronic device and computer readable storage medium

Abstract

This invention provides a decentralized rack awareness method, system, device, and medium based on Ignite in the field of distributed data access technology. The method includes the following steps: Step S10: Calculate the weight of each partition on each node using the highest random weight algorithm, and sort the nodes based on the weights to obtain the node weight order; Step S20: Set the partition held by the node with the highest weight as the primary partition, and set the node holding the primary partition as the primary node; Step S30: Determine the network distance between the remaining nodes and the primary node based on the topology relationship between the host and the nodes; Step S40: Set a threshold for identifying the number of secondary partitions allowed to be held by each rack, and allocate secondary partitions based on the threshold and the network distance; Step S50: Allocate the remaining secondary partitions based on the node weight order. The advantage of this invention is that it greatly improves the reliability and efficiency of data access.

Description

Technical Field

[0001] This invention relates to the field of distributed data access technology, and in particular to a decentralized rack sensing method, system, electronic device, and computer-readable storage medium based on Ignite. Background Technology

[0002] Ignite is a memory-based distributed key-value store that can be viewed as a distributed partitioned hash. In partitioned mode, each node on a host has a fixed number of partitions. The partition where the data will be stored is determined by taking the modulo of the hash value of the data to be stored with the number of partitions. The same partition number can be divided into primary and secondary partitions on different nodes. There is only one primary partition, but multiple secondary partitions are possible. For the same set of data, the data stored in the primary partition is called primary data, and the data stored in the secondary partition is called replica data. Figure 5 As shown.

[0003] The allocation of primary and secondary partitions to specific nodes on different hosts is determined by the Highest Random Weight (HRW) algorithm. This algorithm uses the hash value of the partition number and node information as parameters to calculate the weight of the partition number and each node. Nodes are sorted from highest to lowest weight, with the node with the highest weight holding the primary partition and other nodes having priority for holding the secondary partition.

[0004] However, existing partition distribution technology is host-based, aiming to distribute primary and secondary partitions across nodes on each host. This host-level partition distribution has the following drawbacks: 1. It is possible that the hosts containing the primary and secondary partitions are located in the same rack. When the network or power of the rack is interrupted, all nodes in that rack will go offline, resulting in partition loss and subsequent data loss; 2. The data of the primary and secondary partitions may be distributed in the same or a limited number of racks. When nodes located in other racks access this data, they need to cross hosts and racks, resulting in a larger network distance and thus reducing data access efficiency.

[0005] Therefore, how to provide a decentralized rack sensing method, system, electronic device, and computer-readable storage medium based on Ignite to improve the reliability and efficiency of data access has become an urgent technical problem to be solved. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a decentralized rack sensing method, system, electronic device and computer-readable storage medium based on Ignite, so as to improve the reliability and efficiency of data access.

[0007] In a first aspect, the present invention provides a decentralized rack sensing method based on Ignite, comprising the following steps:

[0008] Step S10: Calculate the weight of each partition in each node using the highest random weight algorithm, and sort the nodes based on the weights to obtain the node weight order;

[0009] Step S20: Set the partition held by the node with the highest weight as the main partition, and set the node holding the main partition as the main node;

[0010] Step S30: Based on the topology relationship between the host and the nodes, determine the network distance between each of the remaining nodes and the master node;

[0011] Step S40: Set a threshold for identifying the number of sub-partitions allowed in each rack, and allocate sub-partitions based on the threshold and network distance;

[0012] Step S50: Allocate the remaining sub-partitions based on the node weight order.

[0013] Furthermore, in step S10, the node is deployed within a host, and the host is deployed within a rack.

[0014] Further, step S30 specifically includes:

[0015] Nodes on the same host as the master node are designated as nearest neighbors, and nodes that are not near neighbors are designated as far neighbors.

[0016] Further, step S40 specifically includes:

[0017] Step S41: Set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, determine whether the network distance of the next node is a distant neighbor. If not, do not allocate a sub-partition to the node and proceed to the next node. If yes, proceed to step S42.

[0018] Step S42: Determine whether the number of secondary partitions already held by the rack corresponding to the current node is less than the threshold number. If yes, allocate a secondary partition to the current node and proceed to the next node determination; otherwise, proceed to the next rack determination.

[0019] Step S43: Determine whether all secondary partitions have been allocated. If not, proceed to step S50; if yes, end the process.

[0020] Secondly, the present invention provides a decentralized rack sensing system based on Ignite, comprising the following modules:

[0021] The node weight calculation module is used to calculate the weight of each partition in each node using the highest random weight algorithm, and sort the nodes based on the weights to obtain the node weight order.

[0022] The primary partition allocation module is used to set the partition held by the node with the highest weight as the primary partition, and to set the node holding the primary partition as the primary node.

[0023] The network distance calculation module is used to determine the network distance between the remaining nodes and the master node based on the topological relationship between the host and the nodes.

[0024] The first sub-partition allocation module is used to set a threshold for identifying the number of sub-partitions allowed in each rack, and to allocate sub-partitions based on the threshold and network distance.

[0025] The second sub-partition allocation module is used to allocate the remaining sub-partitions based on the node weight order.

[0026] Furthermore, in the node weight calculation module, the node is deployed inside the host, and the host is deployed inside the rack.

[0027] Furthermore, the network distance calculation module is specifically used for:

[0028] Nodes on the same host as the master node are designated as nearest neighbors, and nodes that are not near neighbors are designated as far neighbors.

[0029] Furthermore, the first sub-partition allocation module specifically includes:

[0030] The network distance verification unit is used to set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, it determines whether the network distance of the next node is a distant neighbor. If not, the sub-partition is not allocated to the node, and the next node is determined. If so, the process proceeds to the number threshold verification unit.

[0031] The quantity threshold verification unit is used to determine whether the number of sub-partitions already held by the rack corresponding to the current node is less than the quantity threshold. If yes, a sub-partition is allocated to the current node, and the next node is determined; otherwise, the next rack is determined.

[0032] The secondary partition allocation verification unit is used to determine whether the allocation of all secondary partitions has been completed. If not, it proceeds to the second secondary partition allocation module; if yes, the process ends.

[0033] Thirdly, the present invention provides a decentralized rack sensing electronic device based on Ignite, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0034] Fourthly, the present invention provides an Ignite-based decentralized rack-aware computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0035] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0036] The weight of each partition on each node is calculated using the highest random weight algorithm. Based on the weight, the nodes are sorted to obtain the node weight order. The partition held by the node with the highest weight is set as the primary partition, and the node holding the primary partition is set as the primary node. Then, the network distance between the remaining nodes and the primary node is calculated based on the topology relationship between the host and the node. Sub-partitions are allocated based on the set quantity threshold and network distance. Finally, the remaining sub-partitions are allocated based on the node weight order. That is, after allocating the primary partition based on weight, the sub-partitions are allocated to hosts different from the primary partitions, and the number of sub-partitions held by each rack is limited. On this basis, there are still unallocated sub-partitions, which are then allocated based on the node weight order. In other words, the partitions are distributed as widely as possible on different hosts and different racks to avoid partition loss due to network or power failure of a rack, shorten the network distance for data access, and thus greatly improve the reliability and efficiency of data access.

[0037] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] Figure 1 This is a flowchart of a decentralized rack sensing method based on Ignite according to the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of a decentralized rack sensing system based on Ignite according to the present invention.

[0041] Figure 3 This is a schematic diagram of the structure of a decentralized rack sensing electronic device based on Ignite according to the present invention.

[0042] Figure 4 This is a schematic diagram of the structure of a decentralized rack-aware computer-readable storage medium based on Ignite according to the present invention.

[0043] Figure 5 This is a schematic diagram of the allocation of primary and secondary partitions in this invention.

[0044] Figure 6 This is a schematic diagram of the node weight sorting method of the present invention.

[0045] Figure 7 This is a topology diagram of the Ignite cluster of this invention.

[0046] Figure 8 This is a schematic diagram of the partition node filtering method of the present invention. Detailed Implementation

[0047] This application provides a decentralized rack sensing method, system, electronic device, and computer-readable storage medium based on Ignite, thereby improving the reliability and efficiency of data access.

[0048] The technical solution in this application embodiment has the following general idea: after allocating the primary partition based on weight, the secondary partition is allocated to a host different from the primary partition, and the number of secondary partitions held by each rack is limited. On this basis, there are still unallocated secondary partitions, which are then allocated based on the node weight order to avoid partition loss when the network or power of a rack is disconnected, shorten the network distance for data access, and improve the reliability and efficiency of data access. Example 1

[0049] This embodiment provides a decentralized rack awareness method based on Ignite, such as... Figure 1 , 5 As shown in Figures 6, 7, and 8, the steps include the following:

[0050] Step S10: Calculate the weight of each partition on each node based on the Ignite settings using the Highest Random Weight (HWR) algorithm. Sort the nodes from largest to smallest based on the weights to obtain the node weight order, such as... Figure 6 As shown;

[0051] Step S20: Set the partition held by the node with the highest weight as the main partition, and set the node holding the main partition as the main node;

[0052] Step S30: Based on the topology relationship between the host and the nodes, determine the network distance between each of the remaining nodes and the master node;

[0053] Step S40: Set a threshold for identifying the number of sub-partitions allowed in each rack, and allocate sub-partitions based on the threshold and network distance;

[0054] Step S50: Allocate the remaining sub-partitions based on the node weight order.

[0055] Traditionally, all sub-partitions are allocated directly based on the node weight order. However, this invention takes into account both the host and the rack, allowing for a more dispersed distribution of partitions.

[0056] In step S10, the node (server instance process) is deployed within a host, and the host is deployed within a rack; a rack deploys several hosts, and a host deploys several nodes, such as... Figure 7 As shown.

[0057] Step S30 specifically involves:

[0058] Nodes on the same host as the master node are designated as nearest neighbors, and nodes that are not near neighbors are designated as far neighbors.

[0059] Step S40 specifically includes:

[0060] Step S41: Set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, determine whether the network distance of the next node is a distant neighbor. If not, do not allocate a sub-partition to the node and proceed to the next node. If yes, proceed to step S42.

[0061] The formula for calculating the quantity threshold is:

[0062] nodeCount=floor(backupsCount / rackCount);

[0063] Where nodeCount represents the quantity threshold; floor() represents rounding down; backupsCount represents the number of secondary partitions; and rackCount represents the number of racks.

[0064] Step S42: Determine whether the number of secondary partitions already held by the rack corresponding to the current node is less than the threshold number. If yes, allocate a secondary partition to the current node and proceed to the next node determination; otherwise, proceed to the next rack determination.

[0065] like Figure 8As shown, node 1 has the highest weight, so it is assigned a primary partition. All nodes in host 1 are considered neighbors. Node 2 is a neighbor, so it does not hold a secondary partition. Node 3 is a neighbor, so it does not hold a secondary partition. Node 5 is not a neighbor, but it is in the same rack as the neighboring nodes, so it is considered a distant neighbor. It is necessary to check the number of secondary partitions already held by rack 1.

[0066] Step S43: Determine whether all secondary partitions have been allocated. If not, proceed to step S50; if yes, end the process. Example 2

[0067] This embodiment provides a decentralized rack sensing system based on Ignite, such as Figure 2 , 5 As shown in Figures 6, 7, and 8, it includes the following modules:

[0068] The node weight calculation module is used to calculate the weight of each partition on each node based on the Ignite settings using the Highest Random Weight (HWR) algorithm. The nodes are then sorted from largest to smallest based on these weights to obtain the node weight order. Figure 6 As shown;

[0069] The primary partition allocation module is used to set the partition held by the node with the highest weight as the primary partition, and to set the node holding the primary partition as the primary node.

[0070] The network distance calculation module is used to determine the network distance between the remaining nodes and the master node based on the topological relationship between the host and the nodes.

[0071] The first sub-partition allocation module is used to set a threshold for identifying the number of sub-partitions allowed in each rack, and to allocate sub-partitions based on the threshold and network distance.

[0072] The second sub-partition allocation module is used to allocate the remaining sub-partitions based on the node weight order.

[0073] Traditionally, all sub-partitions are allocated directly based on the node weight order. However, this invention takes into account both the host and the rack, allowing for a more dispersed distribution of partitions.

[0074] In the node weight calculation module, the nodes (server instance processes) are deployed within hosts, and the hosts are deployed within racks; a rack deploys several hosts, and a host deploys several nodes, such as... Figure 7 As shown.

[0075] The network distance calculation module is specifically used for:

[0076] Nodes on the same host as the master node are designated as nearest neighbors, and nodes that are not near neighbors are designated as far neighbors.

[0077] The first secondary partition allocation module specifically includes:

[0078] The network distance verification unit is used to set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, it determines whether the network distance of the next node is a distant neighbor. If not, the sub-partition is not allocated to the node, and the next node is determined. If so, the process proceeds to the number threshold verification unit.

[0079] The formula for calculating the quantity threshold is:

[0080] nodeCount=floor(backupsCount / rackCount);

[0081] Where nodeCount represents the quantity threshold; floor() represents rounding down; backupsCount represents the number of secondary partitions; and rackCount represents the number of racks.

[0082] The quantity threshold verification unit is used to determine whether the number of sub-partitions already held by the rack corresponding to the current node is less than the quantity threshold. If yes, a sub-partition is allocated to the current node, and the next node is determined; otherwise, the next rack is determined.

[0083] like Figure 8 As shown, node 1 has the highest weight, so it is assigned a primary partition. All nodes in host 1 are considered neighbors. Node 2 is a neighbor, so it does not hold a secondary partition. Node 3 is a neighbor, so it does not hold a secondary partition. Node 5 is not a neighbor, but it is in the same rack as the neighboring nodes, so it is considered a distant neighbor. It is necessary to check the number of secondary partitions already held by rack 1.

[0084] The secondary partition allocation verification unit is used to determine whether the allocation of all secondary partitions has been completed. If not, it proceeds to the second secondary partition allocation module; if yes, the process ends.

[0085] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3. Example 3

[0086] This embodiment provides a decentralized rack-based sensing electronic device based on Ignite, such as... Figure 3 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the embodiments in Example 1.

[0087] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0088] Based on the same inventive concept, this application provides a computer-readable storage medium corresponding to Embodiment 1, as detailed in Embodiment 4. Example 4

[0089] This embodiment provides a decentralized rack-aware computer-readable storage medium based on Ignite, such as... Figure 4 As shown, a computer program is stored thereon, which, when executed by a processor, can implement any of the embodiments in Example 1.

[0090] Since the storage medium described in this embodiment is the same storage medium used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation methods and various variations of the storage medium in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the storage medium implements the method in this application embodiment will not be described in detail here. Any storage medium used by those skilled in the art to implement the method in this application embodiment falls within the scope of protection of this application.

[0091] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0092] The weight of each partition on each node is calculated using the highest random weight algorithm. Based on the weight, the nodes are sorted to obtain the node weight order. The partition held by the node with the highest weight is set as the primary partition, and the node holding the primary partition is set as the primary node. Then, the network distance between the remaining nodes and the primary node is calculated based on the topology relationship between the host and the node. Sub-partitions are allocated based on the set quantity threshold and network distance. Finally, the remaining sub-partitions are allocated based on the node weight order. That is, after allocating the primary partition based on weight, the sub-partitions are allocated to hosts different from the primary partitions, and the number of sub-partitions held by each rack is limited. On this basis, there are still unallocated sub-partitions, which are then allocated based on the node weight order. In other words, the partitions are distributed as widely as possible on different hosts and different racks to avoid partition loss due to network or power failure of a rack, shorten the network distance for data access, and thus greatly improve the reliability and efficiency of data access.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A decentralized rack sensing method based on Ignite, characterized in that: Includes the following steps: Step S10: Calculate the weight of each partition on each node using the highest random weight algorithm, and sort the nodes based on the weights to obtain the node weight order; the nodes are deployed in the host, and the host is deployed in the rack. Step S20: Set the partition held by the node with the highest weight as the main partition, and set the node holding the main partition as the main node; Step S30: Set the network distance of nodes that are on the same host as the master node as the nearest neighbor, and set the network distance of non-near neighbor nodes as far neighbor. Step S40: Set a threshold for identifying the number of sub-partitions allowed in each rack, and allocate sub-partitions based on the threshold and network distance; Step S50: Allocate the remaining sub-partitions based on the node weight order; Step S40 specifically includes: Step S41: Set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, determine whether the network distance of the next node is a distant neighbor. If not, do not allocate a sub-partition to the node and proceed to the next node. If yes, proceed to step S42. Step S42: Determine whether the number of secondary partitions already held by the rack corresponding to the current node is less than the threshold number. If yes, allocate a secondary partition to the current node and proceed to the next node determination; otherwise, proceed to the next rack determination. Step S43: Determine whether all secondary partitions have been allocated. If not, proceed to step S50; if yes, end the process.

2. A decentralized rack sensing system based on Ignite, characterized in that: Includes the following modules: The node weight calculation module is used to calculate the weight of each partition on each node using the highest random weight algorithm, and sort the nodes based on the weights to obtain the node weight order; the nodes are deployed in the host, and the host is deployed in the rack. The primary partition allocation module is used to set the partition held by the node with the highest weight as the primary partition, and to set the node holding the primary partition as the primary node. The network distance calculation module is used to set the network distance of nodes that are on the same host as the master node as the nearest neighbor, and set the network distance of non-near neighbor nodes as the far neighbor. The first sub-partition allocation module is used to set a threshold for identifying the number of sub-partitions allowed in each rack, and to allocate sub-partitions based on the threshold and network distance. The second sub-partition allocation module is used to allocate the remaining sub-partitions based on the node weight order; The first secondary partition allocation module specifically includes: The network distance verification unit is used to set a threshold for identifying the number of sub-partitions allowed for each rack. Based on the node weight order, it determines whether the network distance of the next node is a distant neighbor. If not, the sub-partition is not allocated to the node, and the next node is determined. If so, the process proceeds to the number threshold verification unit. The quantity threshold verification unit is used to determine whether the number of sub-partitions already held by the rack corresponding to the current node is less than the quantity threshold. If yes, a sub-partition is allocated to the current node, and the next node is determined; otherwise, the next rack is determined. The secondary partition allocation verification unit is used to determine whether the allocation of all secondary partitions has been completed. If not, it proceeds to the second secondary partition allocation module; if yes, the process ends.

3. A decentralized rack-based sensing electronics device based on Ignite, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in claim 1.

4. A decentralized rack-aware computer-readable storage medium based on Ignite, wherein a computer program is stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in claim 1.

Images