Data partitioning method and device in distributed storage, equipment and medium

Abstract

The invention relates to the technical field of data storage, in particular to a data partitioning method and device in distributed storage, equipment and a medium, and the method comprises the following steps: determining a partitioning table corresponding to a primary copy according to codes corresponding to a plurality of nodes and a total partitioning number; and according to the physical node identifier corresponding to each bit number in the partition table corresponding to the master copy, generating a partition table corresponding to the slave copy by adopting a rotating magnetic levitation mode, so that the physical node identifiers corresponding to the same bit number of the partition table corresponding to the slave copy and the partition table corresponding to the master copy are different. Therefore, the master copy and the slave copy of the same data are prevented from being distributed to the same logic node and the same physical node, and data integrity guarantee can be provided when a single physical node is powered down.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and in particular to a data partitioning method, apparatus, device, and medium in distributed storage. Background Technology

[0002] In the rapid development of modern information technology, data storage and management have become crucial areas. With the advent of the big data era, the dramatic increase in data volume has posed enormous challenges to traditional storage solutions. Distributed storage systems, due to their high scalability and fault tolerance, have attracted widespread attention and have become an effective solution for large-scale data storage problems.

[0003] Distributed storage systems can improve data access speed and system reliability to some extent by distributing data across multiple nodes. However, because different copies of data are allocated to different nodes, there is randomness involved, and data loss can easily occur if a physical node experiences a power outage.

[0004] Therefore, how to ensure data integrity in the event of a power outage has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data partitioning method, apparatus, device, and medium in distributed storage that can ensure data integrity in the event of power failure.

[0006] Firstly, a data partitioning method in distributed storage is provided, including:

[0007] Obtain the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs.

[0008] Based on the codes corresponding to each of the multiple nodes and the total number of partitions, the partition table corresponding to the primary replica is determined. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node.

[0009] Based on the physical node identifiers corresponding to each bit in the partition table corresponding to the primary replica, a partition table corresponding to the secondary replica is generated using a rotational magnetic levitation method, such that the physical node identifiers corresponding to the same bit in the partition table corresponding to the secondary replica are different from those in the partition table corresponding to the primary replica.

[0010] In a preferred embodiment, this application can be further configured to: generate a partition table corresponding to a slave replica using a rotational magnetic levitation method based on the physical node identifiers corresponding to each bit in the partition table corresponding to the primary replica, including:

[0011] Rotate the array in the partition table corresponding to the primary replica by a first preset number of bits in a preset direction to obtain the initial partition table corresponding to the secondary replica.

[0012] For the slave replica, determine whether the physical node identifier corresponding to the bit in the initial partition table corresponding to the slave replica is the same as the physical node identifier corresponding to the bit in the partition table corresponding to the master replica; if so, rotate the array after the bit in a preset direction by a second preset bit, until all bits have been traversed to obtain the partition table corresponding to the slave replica.

[0013] If the number of replicas is greater than 2, the array in the partition table corresponding to the replica is rotated by a third preset number of bits in a preset direction to obtain the initial partition table corresponding to the other replicas. For the other replicas, it is determined whether the physical node identifier corresponding to the bit in the initial partition table corresponding to the other replica is the same as the physical node identifier of the corresponding bit in the partition table corresponding to the replica. If so, the array after the bit is rotated by a fourth preset number of bits in a preset direction until all bits are traversed to obtain the partition table corresponding to the other replicas.

[0014] In a preferred example, this application can be further configured to: determine the partition table corresponding to the primary replica based on the respective codes of multiple nodes and the total number of partitions, including:

[0015] Obtain the encoded hash value corresponding to each of the multiple nodes in the cluster. The encoded hash value is determined based on the encoding of the node, and one encoding corresponds to a unique node.

[0016] Seed information is obtained by concatenating the encoded hash value of the target node and the suffix corresponding to the primary replica; based on the seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined; the target node can be any node among multiple nodes.

[0017] Based on the preference sequence results corresponding to each of the multiple nodes, the partitions corresponding to each of the multiple nodes are determined to obtain the partition table corresponding to the primary replica.

[0018] In a preferred embodiment, this application can be further configured such that the preference sequence result includes: offset, step size, displacement value, and total number of partitions;

[0019] Accordingly, based on seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined, including:

[0020] A first digest is calculated on the seed information to obtain a first hash value;

[0021] A second digest is calculated on the seed information to obtain a second hash value;

[0022] The offset of the target node is obtained by taking the modulo between the first hash value and the total number of partitions;

[0023] The step size corresponding to the target node is obtained by calculating the second hash value and the total number of partitions according to a preset formula.

[0024] The preset formula is: Step size = Second hash value mod (total number of partitions - 1) + 1;

[0025] The displacement value is determined based on the offset, step size, and total number of partitions.

[0026] In a preferred embodiment, this application can be further configured to: determine the partitions corresponding to each of the plurality of nodes based on the preference sequence results corresponding to each of the plurality of nodes, including:

[0027] The number of bits is obtained by taking the modulo between the offset corresponding to the target node and the total number of partitions; the target node is any one of the plurality of nodes.

[0028] Update the offset according to the step size, and repeat the modulo operation based on the offset corresponding to the target node and the total number of partitions to obtain the number of bits until the offset and the displacement value are equal, thus obtaining all the bits corresponding to the target node;

[0029] Based on all the bits corresponding to each of the multiple nodes, determine the partition corresponding to each of the multiple nodes.

[0030] In a preferred example, this application can be further configured to: after taking the modulo between the offset corresponding to the target node and the total number of partitions to obtain the number of bits, it also includes:

[0031] If the number of bits corresponding to the target node is the same as the number of bits corresponding to other nodes that precede it, then the number of bits of the target node is ignored.

[0032] In a preferred embodiment, this application may be further configured to include, before determining the partition corresponding to each of the plurality of nodes based on all the bits corresponding to each of the plurality of nodes:

[0033] Determine if there is a target partition without assigned nodes;

[0034] If a target partition exists, determine the number of partitions corresponding to each node; and select the node with the fewest partitions as the node corresponding to the target partition.

[0035] Secondly, a data partitioning device in distributed storage is provided, comprising:

[0036] The acquisition module is used to acquire the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs.

[0037] The primary replica partition table determination module is used to determine the partition table corresponding to the primary replica based on the codes corresponding to multiple nodes and the total number of partitions. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node.

[0038] The partition table determination module is used to generate a partition table corresponding to a slave replica by using a rotational magnetic levitation method based on the physical node identifier corresponding to each bit in the partition table corresponding to the primary replica, so that the physical node identifier corresponding to the same bit in the partition table corresponding to the slave replica is different from that in the partition table corresponding to the primary replica.

[0039] Thirdly, an electronic device is provided, comprising:

[0040] One or more processors;

[0041] Memory;

[0042] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: perform operations corresponding to the methods shown in any possible implementation of the first aspect.

[0043] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded by a processor and performs the steps of the method shown in any possible implementation of the first aspect.

[0044] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements operations corresponding to the methods shown in any possible implementation of the first aspect.

[0045] In summary, the data partitioning method in distributed storage provided in this application has the following beneficial technical effects:

[0046] This application determines the partition table corresponding to the primary replica based on the codes corresponding to multiple nodes and the total number of partitions; based on the physical node identifiers corresponding to each digit in the partition table corresponding to the primary replica, a partition table corresponding to the secondary replica is generated using a rotational magnetic levitation method, so that the physical node identifiers corresponding to the same digit in the partition table corresponding to the secondary replica are different from those corresponding to the primary replica, thereby avoiding the allocation of primary / secondary replicas of the same data on the same logical node and physical node, and providing data integrity guarantee when a single physical node loses power.

[0047] In addition, this application also provides a data partitioning device, equipment and medium in distributed storage, all of which have the above-mentioned beneficial technical effects. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of a data partitioning method in distributed storage provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the structure of a data partitioning device in a distributed storage provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0053] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0056] In distributed storage, research on data partitioning optimization methods mainly focuses on the following aspects: First, how to perform intelligent partitioning based on data access patterns and characteristics to reduce data access latency; second, how to dynamically adjust data partitioning strategies to cope with changes in data volume and the dynamic addition or removal of nodes; and finally, how to maximize system reliability while ensuring data consistency and avoid data loss due to service power outages.

[0057] While various data partitioning methods have been proposed and applied in existing distributed storage technologies, some significant drawbacks still limit their effectiveness in large-scale data environments.

[0058] Many traditional data partitioning methods rely on static partitioning strategies, meaning that the system cannot adjust the partitioning scheme in a timely manner when data volume or access patterns change. This static nature leads to the data hotspot problem, where some nodes become overloaded due to storing too much popular data, while other nodes remain idle, resulting in wasted resources and performance degradation.

[0059] Related technologies often face challenges in handling data consistency. Especially in dynamic environments, the joining and leaving of nodes can lead to uneven data partitioning, thereby affecting data availability and consistency. Many methods, while ensuring data consistency, often sacrifice performance, resulting in increased system response time.

[0060] Existing technologies do not consider physical node factors when partitioning data, so they cannot allocate different copies of data to different physical nodes in order to maximize data integrity in the event of a node power failure.

[0061] Therefore, a new data partitioning optimization method is proposed, which can not only improve the performance of distributed storage systems, but also provide a more efficient solution for large-scale data processing.

[0062] To address the shortcomings of related methods, this invention proposes a data partitioning optimization method in distributed storage. By using a rotating magnetic levitation method, the dynamic and random nature of data partitioning is ensured. Furthermore, by configuring physical node information during data partitioning calculation, more reliable data integrity is provided.

[0063] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application are explained, and the scenarios and terms involved in the embodiments of this application are explained as follows:

[0064] Distributed storage clusters consist of multiple nodes;

[0065] Multiple nodes correspond to several physical nodes. For example, node 1 and node 2 correspond to the same physical node a, and node 3 corresponds to a physical node b.

[0066] Set up multiple data partitions, each partition corresponds to a node, and a node may correspond to multiple partitions. For example, node 1 corresponds to partition 1, partition 3, and partition 5, and node 2 corresponds to partition 2, partition 4, and partition 6.

[0067] In this embodiment of the application, the following parameters are defined for ease of understanding:

[0068] The total number of data partitions is PNum;

[0069] The system data replica count is repl;

[0070] The total number of nodes in the cluster is CNum;

[0071] The preference sequence array to be generated is Q[i], with offset, step size, and shift value;

[0072] Each node in the cluster is coded as memberId;

[0073] The hash value encoded by the cluster node is nodeHash;

[0074] Each node is configured with a physical node ID called rackId.

[0075] The following is a detailed description. This application provides a data partitioning method in distributed storage, such as... Figure 1 As shown, the method provided in this application embodiment can be executed by an electronic device, which is a server. This server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and electronic device can be directly or indirectly connected via wired or wireless communication. This application embodiment does not impose any limitations on this connection. The method includes:

[0076] S101. Obtain the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs.

[0077] In this embodiment, the encoding corresponding to each node is determined based on the IP address and port number of each node in the cluster. The encoded memberId is obtained by concatenating the node's IP address and port number, ensuring that each node's memberId is unique.

[0078] Furthermore, prior to S101, it also includes: determining the physical node identifier of each of the multiple nodes based on the physical node information of each of the multiple nodes in the cluster, wherein if the physical node information of two nodes is the same, then the physical node identifier is configured to be the same.

[0079] The encoding corresponding to each of the multiple nodes in the cluster is determined based on the IP address and port number of the node.

[0080] In this embodiment, the physical node identifier `rackId` is configured based on the physical node information of each node in the cluster. If two nodes have the same physical node information, the same `rackId` is configured; otherwise, different `rackId`s are configured. The node's encoded `memberId` is configured, which can be achieved by concatenating the node's IP address and port number, ensuring that each node in the cluster has a unique `memberId`.

[0081] S102. Based on the codes corresponding to each of the multiple nodes and the total number of partitions, determine the partition table corresponding to the primary replica. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node.

[0082] S103. Based on the physical node identifier corresponding to each bit in the partition table corresponding to the primary replica, generate the partition table corresponding to the secondary replica using a rotational magnetic levitation method, so that the physical node identifier corresponding to the same bit in the partition table corresponding to the secondary replica is different from that in the partition table corresponding to the primary replica.

[0083] In this embodiment, the primary copy is rotated according to a preset rotation direction and a preset number of bits to obtain the partition table corresponding to the secondary copy. A rotational magnetic levitation method is used to generate at least one partition table for each copy, ensuring that different copies of data are distributed across different nodes to the greatest extent possible while maintaining node randomness.

[0084] As can be seen, in this embodiment, the partition table corresponding to the primary replica is determined based on the codes corresponding to each of the multiple nodes and the total number of partitions; based on the physical node identifiers corresponding to each digit in the partition table corresponding to the primary replica, a partition table corresponding to the secondary replica is generated using a rotating magnetic levitation method, so that the physical node identifiers corresponding to the same digit in the partition table corresponding to the secondary replica are different from those corresponding to the primary replica, so as to avoid the primary / secondary replicas of the same data being allocated to the same logical node and physical node, and to provide data integrity guarantee when a single physical node loses power.

[0085] One possible implementation of this application embodiment is as follows: S103 generates a partition table corresponding to a slave replica using a rotational magnetic levitation method based on the physical node identifiers corresponding to each bit in the partition table corresponding to the primary replica.

[0086] S1031. Rotate the array in the partition table corresponding to the primary replica by a first preset number of bits in a preset direction to obtain the initial partition table corresponding to the secondary replica.

[0087] In this embodiment of the application, the array is a one-dimensional array, and the preset direction can be left or right. The first preset number of bits can be set by the user according to actual needs, and can be 1 or 2 bits.

[0088] For example, the partition table p0 corresponding to the primary replica is a one-dimensional array with a length equal to the total number of partitions PNum. The values ​​of the array are the indices calculated for each partition, i.e., p0 is in the form [0,1,0,2,1,1,...,0,2,1]. Rotating p0 one position to the right yields a new array, i.e., the initial partition table p0_1, which is in the form [1,0,2,1,1,...,0,2,1,0].

[0089] S1032. For the slave replica, determine whether the physical node identifier corresponding to the bit in the initial partition table corresponding to the slave replica is the same as the physical node identifier corresponding to the bit in the partition table corresponding to the master replica; if so, rotate the array after the bit in the preset direction by a second preset bit, until all bits have been traversed to obtain the partition table corresponding to the slave replica.

[0090] In one feasible approach, it is sequentially determined whether the physical node identifier corresponding to the number of bits in the initial partition table is the same as the physical node identifier of the corresponding number of bits in the partition table corresponding to the primary replica, with each value corresponding to a specific node. If so, the array following the number of bits is rotated by a preset number of bits in a preset direction until all bits have been traversed. It is understood that the first preset number of bits and the second preset number of bits can be the same or different.

[0091] In another possible implementation, the values ​​corresponding to the bits in the initial partition table are sequentially checked against the corresponding bits in the partition table of the primary replica, with each value corresponding to a node. If they are, the array after the bits is rotated by a second preset number of bits in a preset direction until all bits have been traversed, resulting in a second initial partition table. The physical node identifiers corresponding to the bits in the second initial partition table are then sequentially checked against the corresponding physical node identifiers in the partition table of the primary replica, with each value corresponding to a node. If they are, the array after the bits is rotated by a second preset number of bits in a preset direction until all bits have been traversed. For example, it is sequentially checked whether the index of each element in p0_1 is the same as that of p0. If they are, the element and all subsequent elements are rotated one position to the right. The rackId corresponding to the index of the element is then checked whether it is the same as that of p0. If they are, the element and all subsequent elements are also rotated one position to the right. This process is repeated until all elements in p0_1 have been traversed, resulting in p1.

[0092] S1033. If the number of replicas is greater than 2, the array in the partition table corresponding to the replica will be rotated by a third preset number of positions in a preset direction to obtain the other initial partition tables corresponding to the replicas.

[0093] S1034. For other slave replicas, determine in turn whether the physical node identifier corresponding to the bit in the initial partition table of the other slave replica is the same as the physical node identifier of the corresponding bit in the partition table of the slave replica; if so, rotate the array after the bit in the preset direction by the fourth preset bit, until the traversal of all bits is completed to obtain the partition table corresponding to the other slave replica.

[0094] The first preset number of digits, the second preset number of digits, the third preset number of digits, and the fourth preset number of digits can be exactly the same or not exactly the same.

[0095] As can be seen, in this embodiment, a rotation method is used to generate partition tables for multiple replicas. The secondary replica partition table is obtained by rotating the primary replica partition table, and other secondary replica partition tables are obtained by rotating the secondary replica partition table, eliminating the need to repeatedly calculate partition table information. During the calculation of the data partition table for each replica, identical logical and physical nodes are filtered out, preventing the primary / secondary replicas of the same data from being assigned to the same logical and physical nodes, thus improving reliability.

[0096] One possible implementation of this application embodiment, S102, involves determining the partition table corresponding to the primary replica based on the respective codes of multiple nodes and the total number of partitions, including:

[0097] S1021. Obtain the encoded hash value corresponding to each of the multiple nodes in the cluster. The encoded hash value is determined based on the encoding of the node, and one encoding corresponds to a unique node.

[0098] In some embodiments, a hash algorithm can be used to perform a hash operation on the encoding to obtain the encoded hash value. The specific hash algorithm used is not limited in this application embodiment; MD5, SHA-1, SHA-256, etc., can be used. For example, based on the memberId of each node in the cluster, a 32-bit hash operation is performed to obtain the hash result of each node's encoding, i.e., the encoded hash value nodeHash.

[0099] S1022. Concatenate the encoded hash value of the target node and the suffix corresponding to the primary replica to obtain the seed information; based on the seed information and the total number of partitions, determine the preference sequence result corresponding to the target node; the target node can be any node among multiple nodes;

[0100] The number of replicas is not limited in this embodiment of the application. Replicas may include primary replicas and secondary replicas. Each replica corresponds to a replica number REPL, for example, the primary replica corresponds to replica number 0, the first secondary replica corresponds to replica number 1, and the second secondary replica corresponds to replica number 2.

[0101] In this embodiment, the number of replicas is traversed. If the number of replicas corresponds to the number of replicas of the primary replica, the suffix is ​​determined to be the first character; otherwise, the suffix is ​​determined to be the second character. This embodiment does not limit the first and second characters, as long as they are different. For example, the first character is M and the second character is S.

[0102] Next, the encoded hash value nodeHash in the cluster node is concatenated with the suffix of the primary replica to obtain the seed information. Then, the seed information and the total number of partitions PNum are combined to generate the preference sequence result Q[i] corresponding to the target node, where i is the target cluster. Through this step, the preference sequence results corresponding to all nodes can be obtained.

[0103] S1023. Based on the preference sequence results of each of the multiple nodes, determine the partitions corresponding to each of the multiple nodes to obtain the partition table corresponding to the primary replica.

[0104] In this embodiment of the application, a partition table p0 with primary replica repl=0 is generated based on the calculated preference sequence results of each node.

[0105] For example, the partition table p0=[0,1,0,2,1,1,0,2,1] has 9 digits. It can be seen that the cluster has three nodes and 9 partitions. The nodes corresponding to the 9 partitions are: 0, 1, 0, 2, 1, 1, 0, 2, 1.

[0106] As can be seen, in this embodiment of the application, when determining the partition table of the primary replica, the seed information is obtained by concatenating the encoded hash value of the target node and the suffix corresponding to the primary replica; based on the seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined; based on the preference sequence result corresponding to each of the multiple nodes, the partition corresponding to each of the multiple nodes is determined to obtain the partition table corresponding to the primary replica, thus ensuring the dynamicness, randomness and balance of data partitioning.

[0107] One possible implementation of this application embodiment is that the preference sequence result includes: offset, step size, displacement value, and total number of partitions;

[0108] Accordingly, based on seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined, including:

[0109] Calculate the first digest of the seed information to obtain the first hash value;

[0110] A second digest is calculated from the seed information to obtain the second hash value;

[0111] The offset of the target node is obtained by taking the modulo between the first hash value and the total number of partitions;

[0112] The step size corresponding to the target node is obtained by calculating the second hash value and the total number of partitions according to a preset formula.

[0113] The preset formula is: Step size = Second hash value mod (total number of partitions - 1) + 1;

[0114] The displacement value is determined based on the offset, step size, and total number of partitions.

[0115] In this application embodiment, the two digest calculation algorithms can be the same or different, and the user can set them according to actual needs. In some feasible methods, the SHA_256 digest algorithm is used to generate the offset hash value of the seed information, i.e., the first hash value, and the MD5 digest algorithm is used to generate the step hash value of the seed, i.e., the second hash value.

[0116] Specifically, the first hash value is modulo the total number of partitions PNum to obtain the offset of the preference sequence;

[0117] The second hash value is modulo PNum-1 and then incremented by 1 to obtain the step size of the preference sequence.

[0118] Using the offset and step, the shift value of the preference sequence is calculated as shift = offset + step * PNum.

[0119] The offset, step, total number of partitions PNum, and shift are used as the four attribute values ​​of the preference sequence result Q[i].

[0120] One possible implementation of this application embodiment involves determining the partitions corresponding to each of the multiple nodes based on the preference sequence results corresponding to each node, including:

[0121] The number of bits is obtained by taking the modulo between the offset corresponding to the target node and the total number of partitions; the target node can be any one of multiple nodes.

[0122] Update the offset according to the step size, and repeat the modulo operation based on the offset corresponding to the target node and the total number of partitions to obtain the number of bits until the offset and the displacement value are equal, thus obtaining all the bits corresponding to the target node;

[0123] Based on all the bits corresponding to each of the multiple nodes, determine the partition corresponding to each of the multiple nodes.

[0124] Specifically, starting from 0, the cluster's node count CNum is traversed. When offset is not equal to shift, the node index (number of bits) corresponding to the partition is obtained by performing a modulo operation of offset % PNum. Here, partition k = offset % PNum, and this number of bits represents associating the node with the partition corresponding to the number of bits. At the same time, offset is set to offset + step, i.e., offset = offset + step. Then, the step of "obtaining the node index corresponding to the partition by performing a modulo operation of offset % PNum" is repeated until offset equals shift.

[0125] Furthermore, during the partition allocation process for a certain node, if the number of bits corresponding to the target node is the same as the number of bits corresponding to other nodes that came before, then the number of bits of the target node is ignored, or the previous allocation is canceled and the partition is allocated to the current node. Users can choose according to their actual needs, and this application embodiment does not limit it.

[0126] One possible implementation of this application embodiment, before determining the partition corresponding to each of the multiple nodes based on all the bits corresponding to each of the multiple nodes, further includes:

[0127] Determine if there is a target partition without an assigned node; if there is a target partition, determine the number of partitions corresponding to each node; and select the node with the fewest partitions as the node corresponding to the target partition.

[0128] When all traversals are complete, if there are partitions that have not been assigned any nodes, they are filled using the index of the node with the fewest allocations. Furthermore, for partitions that cannot be allocated, the principle of least allocation is used to assign them to nodes, ensuring balanced data distribution.

[0129] The following describes a data partitioning device in distributed storage provided by an embodiment of this application. The data partitioning device in distributed storage described below can be referred to in correspondence with the data partitioning method in distributed storage described above. The data partitioning device 200 in distributed storage of this embodiment is installed in an electronic device. Figure 2 , Figure 2 This is a structural block diagram of a data partitioning device in a distributed storage according to one embodiment of this application, comprising:

[0130] The acquisition module 210 is used to acquire the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs.

[0131] The primary replica partition table determination module 220 is used to determine the partition table corresponding to the primary replica based on the codes corresponding to multiple nodes and the total number of partitions. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node.

[0132] The replica partition table determination module 230 is used to generate a replica partition table based on the physical node identifier corresponding to each bit in the partition table corresponding to the primary replica, using a rotational magnetic levitation method, so that the physical node identifier corresponding to the same bit in the replica partition table is different from that in the primary replica partition table.

[0133] In one feasible approach, module 230 is determined from the replica partition table, specifically for:

[0134] Rotate the array in the partition table corresponding to the primary replica by a preset number of bits in a preset direction to obtain the initial partition table corresponding to the secondary replica.

[0135] For the slave replica, determine whether the physical node identifier corresponding to the bit in the initial partition table of the slave replica is the same as the physical node identifier of the corresponding bit in the partition table of the master replica; if so, rotate the array after the bit in the preset direction by a second preset bit, until all bits have been traversed to obtain the partition table corresponding to the slave replica.

[0136] If the number of replicas is greater than 2, the array in the partition table corresponding to the replica will be rotated by a third preset number of bits in a preset direction to obtain the initial partition table corresponding to the other replicas. For the other replicas, it will be determined whether the physical node identifier corresponding to the bit in the initial partition table corresponding to the other replica is the same as the physical node identifier of the corresponding bit in the partition table corresponding to the replica. If so, the array after the bit will be rotated by a fourth preset number of bits in a preset direction until all bits have been traversed to obtain the partition table corresponding to the other replicas.

[0137] In one feasible implementation, the primary replica partition table determination module 220 is also used for:

[0138] Obtain the encoded hash value corresponding to each of the multiple nodes in the cluster. The encoded hash value is determined based on the encoding of the node, and one encoding corresponds to a unique node.

[0139] Seed information is obtained by concatenating the encoded hash value of the target node and the suffix corresponding to the primary replica; based on the seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined; the target node can be any node among multiple nodes.

[0140] Based on the preference sequence results of multiple nodes, the partitions corresponding to each node are determined to obtain the partition table corresponding to the primary replica.

[0141] In one feasible approach, the preference sequence results include: offset, step size, displacement value, and total number of partitions;

[0142] Correspondingly, the primary / replica partition table determination module 220 is also used for:

[0143] Calculate the first digest of the seed information to obtain the first hash value;

[0144] A second digest is calculated from the seed information to obtain the second hash value;

[0145] The offset of the target node is obtained by taking the modulo between the first hash value and the total number of partitions;

[0146] The step size corresponding to the target node is obtained by calculating the second hash value and the total number of partitions according to a preset formula.

[0147] The preset formula is: Step size = Second hash value mod (total number of partitions - 1) + 1;

[0148] The displacement value is determined based on the offset, step size, and total number of partitions.

[0149] In one feasible implementation, the primary replica partition table determination module 220 is also used for:

[0150] The number of bits is obtained by taking the modulo between the offset corresponding to the target node and the total number of partitions; the target node can be any one of multiple nodes.

[0151] Update the offset according to the step size, and repeat the modulo operation based on the offset corresponding to the target node and the total number of partitions to obtain the number of bits until the offset and the displacement value are equal, thus obtaining all the bits corresponding to the target node;

[0152] Based on all the bits corresponding to each of the multiple nodes, determine the partition corresponding to each of the multiple nodes.

[0153] In one feasible implementation, the primary replica partition table determination module 220 is also used for:

[0154] If the number of bits corresponding to the target node is the same as the number of bits corresponding to other nodes that precede it, then the number of bits of the target node is ignored.

[0155] In one feasible implementation, the primary replica partition table determination module 220 is also used for:

[0156] Determine if there is a target partition without assigned nodes;

[0157] If a target partition exists, determine the number of partitions corresponding to each node; and select the node with the fewest partitions as the node corresponding to the target partition.

[0158] This application provides an electronic device, such as... Figure 3 As shown, Figure 3 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0159] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0160] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0161] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0162] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0163] Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0164] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0165] This application provides a computer program product, including a computer program that, when executed by a processor, implements the corresponding content in the aforementioned method embodiments.

[0166] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0167] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data partitioning method in distributed storage, characterized in that, include: Obtain the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs. Based on the codes corresponding to each of the multiple nodes and the total number of partitions, the partition table corresponding to the primary replica is determined. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node. Based on the physical node identifiers corresponding to each bit in the partition table corresponding to the primary replica, a partition table corresponding to the secondary replica is generated using a rotational magnetic levitation method, such that the physical node identifiers corresponding to the same bit in the partition table corresponding to the secondary replica are different from those in the partition table corresponding to the primary replica.

2. The data partitioning method in distributed storage according to claim 1, characterized in that, Based on the physical node identifiers corresponding to each bit in the partition table corresponding to the primary replica, a partition table corresponding to the secondary replica is generated using a rotational magnetic levitation method, including: Rotate the array in the partition table corresponding to the primary replica by a first preset number of bits in a preset direction to obtain the initial partition table corresponding to the secondary replica. For the slave replica, determine whether the physical node identifier corresponding to the bit in the initial partition table corresponding to the slave replica is the same as the physical node identifier corresponding to the bit in the partition table corresponding to the master replica; if so, rotate the array after the bit in a preset direction by a second preset bit, until all bits have been traversed to obtain the partition table corresponding to the slave replica. If the number of replicas is greater than 2, the array in the partition table corresponding to the replica will be rotated by a preset number of positions in a preset direction to obtain the initial partition table corresponding to the other replicas. For the other replicas, it will be determined whether the physical node identifier corresponding to the position in the initial partition table corresponding to the other replica is the same as the physical node identifier of the corresponding position in the partition table corresponding to the replica. If so, the array after the position will be rotated by a preset number of positions in a preset direction to obtain the partition table corresponding to the other replicas.

3. The data partitioning method in distributed storage according to claim 1, characterized in that, Based on the respective codes of multiple nodes and the total number of partitions, determine the partition table corresponding to the primary replica, including: Obtain the encoded hash value corresponding to each of the multiple nodes in the cluster. The encoded hash value is determined based on the encoding of the node, and one encoding corresponds to a unique node. Seed information is obtained by concatenating the encoded hash value of the target node and the suffix corresponding to the primary replica; based on the seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined; the target node can be any node among multiple nodes. Based on the preference sequence results corresponding to each of the multiple nodes, the partitions corresponding to each of the multiple nodes are determined to obtain the partition table corresponding to the primary replica.

4. The data partitioning method in distributed storage according to claim 3, characterized in that, The preference sequence results include: offset, step size, displacement value, and total number of partitions; Accordingly, based on seed information and the total number of partitions, the preference sequence result corresponding to the target node is determined, including: A first digest is calculated on the seed information to obtain a first hash value; A second digest is calculated on the seed information to obtain a second hash value; The offset of the target node is obtained by taking the modulo between the first hash value and the total number of partitions; The step size corresponding to the target node is obtained by calculating the second hash value and the total number of partitions according to a preset formula. The preset formula is: Step size = Second hash value mod (total number of partitions - 1) + 1; The displacement value is determined based on the offset, step size, and total number of partitions.

5. The data partitioning method in distributed storage according to claim 3, characterized in that, Based on the preference sequence results corresponding to each of the multiple nodes, the partitions corresponding to each of the multiple nodes are determined, including: The number of bits is obtained by taking the modulo between the offset corresponding to the target node and the total number of partitions; the target node is any one of the plurality of nodes. Update the offset according to the step size, and repeat the modulo operation based on the offset corresponding to the target node and the total number of partitions to obtain the number of bits until the offset and the displacement value are equal, thus obtaining all the bits corresponding to the target node; Based on all the bits corresponding to each of the multiple nodes, determine the partition corresponding to each of the multiple nodes.

6. The data partitioning method in distributed storage according to claim 5, characterized in that, After taking the modulo between the offset corresponding to the target node and the total number of partitions to obtain the number of bits, the following is also included: If the number of bits corresponding to the target node is the same as the number of bits corresponding to other nodes that precede it, then the number of bits of the target node is ignored.

7. The data partitioning method in distributed storage according to claim 5, characterized in that, Before determining the partition corresponding to each of the plurality of nodes based on all the bits corresponding to each of the plurality of nodes, the process further includes: Determine if there is a target partition without assigned nodes; If a target partition exists, determine the number of partitions corresponding to each node; and select the node with the fewest partitions as the node corresponding to the target partition.

8. A data partitioning device in distributed storage, characterized in that, include: The acquisition module is used to acquire the codes corresponding to multiple nodes in the cluster, the physical node identifiers corresponding to multiple nodes, and the total number of partitions. The physical node identifiers represent the physical node information to which the node belongs. The primary replica partition table determination module is used to determine the partition table corresponding to the primary replica based on the codes corresponding to multiple nodes and the total number of partitions. Each bit in the partition table corresponds to a partition, and the value of each bit corresponds to a node. The partition table determination module is used to generate a partition table corresponding to a slave replica by using a rotational magnetic levitation method based on the physical node identifier corresponding to each bit in the partition table corresponding to the primary replica, so that the physical node identifier corresponding to the same bit in the partition table corresponding to the slave replica is different from that in the partition table corresponding to the primary replica.

9. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to: perform the steps of the data partitioning method in distributed storage according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded by a processor and executed according to the steps of the data partitioning method in the distributed storage according to any one of claims 1 to 7.

Images