Data distribution method and device based on consistent Hash and related equipment

Abstract

The invention relates to the technical field of data storage, and discloses a data distribution method and device based on consistent Hash and related equipment, and the data distribution method based on consistent Hash comprises the following steps: determining copy weights of a plurality of copy sets by utilizing the storage performance of the copy sets; interval division is carried out on the consistency hash ring based on the replica weight, interval intervals of the replica set are obtained, and the larger the replica weight is, the larger the interval intervals are; calculating a hash value of the object data based on the hash function, and mapping the object data to a consistent hash ring; and searching object data in the interval of the copy set in the consistent hash ring, and storing the object data to the copy set. The problem of unbalanced read-write resource utilization rate can be solved, the read-write performance of the object storage system is improved, and the space load balancing effect is optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a data allocation method, device and related equipment based on consistent hashing. Background Art

[0002] Distributed storage system is to store data in multiple independent devices. Most traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, which cannot meet the needs of large-scale storage applications. Distributed network storage system adopts a scalable system structure, uses multiple storage servers to share storage load, and uses location servers to locate storage information. It improves the reliability, availability and access efficiency of the system to a certain extent.

[0003] In recent years, the era of big data has continuously generated a large number of small files. In the process of distributed storage, it is easy to cause problems such as uneven distribution of copies and large amounts of random reads and writes, resulting in data access delays and performance degradation. The organization of small file objects in object storage systems has problems such as unbalanced layout and uneven loading.

[0004] Currently, the layout algorithm of general object storage systems is designed based on hierarchical and recursive object placement rules, and uses consistent hashing methods to evenly distribute data in various storage units of the object storage system. However, the uniform distribution of data will lead to an imbalance between the performance of the storage unit and data loading, resulting in uneven utilization of read and write resources, affecting the performance of the object storage system. Summary of the invention

[0005] In view of this, the present invention provides a data allocation method, device and related equipment based on consistent hashing to solve the problem of unbalanced read and write resource utilization, improve the read and write performance of the object storage system, and optimize the spatial load balancing effect.

[0006] In a first aspect, the present invention provides a data distribution method based on consistent hashing, which is applied to an object storage system, wherein the object storage system includes multiple replica sets. The data distribution method based on consistent hashing includes: determining the replica weights of the multiple replica sets using the replica set storage performance; dividing the consistent hash ring into intervals based on the replica weights to obtain the intervals of the replica set, wherein the larger the replica weight, the larger the interval; calculating the hash value of the object data based on a hash function, and mapping the object data to the consistent hash ring; searching for the object data within the interval of the replica set in the consistent hash ring, and storing the object data in the replica set.

[0007] In this implementation, by considering the storage performance differences of replica sets, a weight is assigned to each replica set, and based on the consistent hashing method, the replica sets are divided into intervals according to the replica weights. The larger the replica weight, the larger the interval, and the object data is evenly distributed in the interval, so that the replica set with a larger weight stores more object data. It can balance the storage performance of the replica set and data loading, solve the problem of uneven utilization of read and write resources, improve the read and write performance of the object storage system, optimize the space load balancing effect, and improve the efficiency of random access.

[0008] In an optional implementation, the replica set storage performance includes a CPU status parameter, a memory status parameter, a disk status parameter, and a network status parameter of each replica set.

[0009] In an optional implementation, the consistent hash ring is divided into intervals based on the replica weights to obtain the intervals of the replica set, including: dividing the consistent hash ring into intervals according to the ratio of the replica weights of multiple replica sets to obtain the intervals corresponding to each replica set; and mapping the replica set to the corresponding interval using a hash function.

[0010] In this implementation, the consistent hashing method is combined to perform interval division on the consistent hashing ring, thereby improving the efficiency of interval division and facilitating the data allocation method combined with the consistent hashing to achieve efficient data allocation and storage.

[0011] In an optional embodiment, each replica set includes multiple storage nodes, and after searching for object data within the interval of the replica set in the consistent hash ring and mapping the object data to the replica set, it also includes: determining the node weights of the multiple storage nodes based on the storage performance of the storage nodes; for a replica set, dividing all object data according to the node weights of the storage nodes, and storing the object data in the corresponding storage nodes.

[0012] In this implementation, data is further stored in the replica set according to the performance of the storage nodes. When objects are stored, they are organized at two levels: the replica set and the storage nodes in the replica set. This can meet the fault tolerance of object storage and further balance the storage performance and data loading of the storage nodes.

[0013] In an optional embodiment, searching for object data within the interval of the replica set in the consistent hash ring and mapping the object data to the replica set also includes: when a new replica set is added, redetermining the replica weight of the replica set based on the storage performance of the new replica set; determining a new interval of the old replica set on the consistent hash ring based on the replica weight; determining a redundant interval between the new interval and the old interval; and migrating the object data within the redundant interval from the old replica set to the new replica set.

[0014] In this implementation, the storage scenario of increasing replica sets is taken into consideration, and the intervals are reallocated according to the replica weights. The object data in the new interval of the old replica set is not moved, and only the object data in the redundant interval is migrated to the new replica set. Adaptive migration objects are added according to the scale, and data migration does not occur between existing replica sets. This can reduce the amount of data migration, thereby reducing the system performance jitter caused by data migration and optimizing the spatial load balancing effect.

[0015] In an optional embodiment, searching for object data within the interval of the replica set in the consistent hash ring and mapping the object data to the replica set also includes: when deleting the replica set, redetermining the replica weight of the replica set based on the storage performance of the remaining replica sets; determining a new interval of the remaining replica sets on the consistent hash ring based on the replica weight; determining an extension interval between the new interval and the old interval; and migrating the object data in the deleted replica set to the remaining replica sets according to the extension interval.

[0016] In this implementation, the storage scenario of replica set deletion is considered, the interval is reallocated according to the replica weight, the object data in the old interval of the old replica set is not moved, only the object data in the interval of the deleted replica set is migrated to each remaining replica set, and the adaptive migration objects are added according to the scale. Data migration will not occur between existing replica sets, which can reduce the amount of data migration and thus reduce the system performance jitter caused by data migration, and optimize the spatial load balancing effect.

[0017] In the second aspect, the present invention provides a data distribution device based on consistent hashing, which includes: a weight calculation module, which is used to determine the replica weights of multiple replica sets using the replica set storage performance; an interval division module, which is used to divide the consistent hash ring into intervals based on the replica weights to obtain the interval intervals of the replica set, and the larger the replica weight, the larger the interval interval; a data mapping module, which is used to calculate the hash value of the object data based on the hash function, and map the object data to the consistent hash ring; a data storage module, which is used to search for object data within the interval interval of the replica set in the consistent hash ring, and store the object data in the replica set.

[0018] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the consistent hashing-based data allocation method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the consistent hashing-based data allocation method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the data allocation method based on consistent hashing according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flow chart of a data allocation method based on consistent hashing according to an embodiment of the present invention;

[0023] Figure 2 is a flow chart of another data allocation method based on consistent hashing according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of an object data storage method based on a consistent hash ring according to an embodiment of the present invention;

[0025] Figure 4 is a flow chart of a data allocation method under a change in the scale of an object storage system according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of object data migration according to an embodiment of the present invention;

[0027] Figure 6 is a flow chart of another data allocation method under the change of object storage system scale according to an embodiment of the present invention;

[0028] Figure 7 is another schematic diagram of object data migration according to an embodiment of the present invention;

[0029] Figure 8 is a structural block diagram of a data distribution device based on consistent hashing according to an embodiment of the present invention;

[0030] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] Object storage is a technology used to handle data storage, storing data in the form of objects instead of traditional file or block storage. Object storage systems process data as objects, each of which includes the data itself, variable metadata, and a globally unique identifier.

[0033] The object storage system consists of multiple replica sets, each of which is the smallest unit of logical storage of object data. There is a many-to-one mapping relationship between object data and replica sets.

[0034] Among them, the replica set is composed of multiple storage nodes to meet the fault tolerance of object storage. Data synchronization is used between storage nodes to achieve redundant storage of object data.

[0035] According to an embodiment of the present invention, an embodiment of a data distribution method based on consistent hashing is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0036] In this embodiment, a data distribution method based on consistent hashing is provided, which can be applied to the above-mentioned object storage system. The consistent hashing data distribution method distributes object data in a replica set. Figure 1 is a flow chart of a data distribution method based on consistent hashing according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 1 The process sequence shown is limited.

[0037] like Figure 1 As shown, the process includes the following steps:

[0038] Step S101, determining the replica weights of multiple replica sets using the replica set storage performance.

[0039] Build a comprehensive performance evaluation model for each replica set in the object storage system, obtain storage performance data for each replica set, and determine the replica weight of each replica set based on the comprehensive performance evaluation model.

[0040] Among them, the better the storage performance of the replica set, the greater the replica weight of the replica set.

[0041] Step S102, divide the consistent hash ring into intervals based on the replica weights to obtain intervals of the replica set.

[0042] Among them, the consistent hash ring is a data partitioning algorithm for distributed systems. The consistent hash algorithm is used to map the entire hash value space into a virtual ring, making the mapping relationship between data and nodes more stable and balanced.

[0043] The consistent hash ring is divided into intervals according to the ratio of the replica weights of multiple replica sets to obtain the intervals corresponding to each replica set. The larger the replica weight, the larger the interval, and the intervals of each replica set do not overlap.

[0044] According to the interval corresponding to the replica set, the replica set is mapped on the consistent hash ring using a hash function algorithm.

[0045] For example, an object storage system includes three replica sets, and the replica weights of the three replica sets are 0.6, 1, and 0.4 respectively, among which the replica weight of the first replica set is 3 / 10, the replica weight of the second replica set is 1 / 2, and the replica weight of the third replica set is 1 / 5. Therefore, the 3 / 10 area of ​​the consistent hash ring is used as the interval interval of the first replica set, the 1 / 2 area of ​​the consistent hash ring is used as the interval interval of the second replica set, and the 1 / 5 area of ​​the consistent hash ring is used as the interval interval of the third replica set. The three interval intervals constitute a complete consistent hash ring.

[0046] Construct a hash function algorithm and use the hash function algorithm to map each replica set into the corresponding interval.

[0047] In one implementation, a hash function algorithm is used to map each replica set to the intersection of the corresponding interval and the adjacent interval. Specifically, a hash function algorithm is used to map each replica set to the intersection of the corresponding interval and the clockwise adjacent interval.

[0048] Step S103, calculating a hash value of the object data based on a hash function, and mapping the object data to a consistent hash ring.

[0049] Construct a hash function algorithm, map the object data to the interval of the hash space, obtain the hash value of each object data, and map the object data to the consistent hash ring.

[0050] Step S104: searching the consistent hash ring for object data within the interval of the replica set, and storing the object data in the replica set.

[0051] The object data mapped on the consistent hash ring is distributed in multiple intervals, and the object data in each interval is the data of the corresponding replica set, and the object data is stored in the corresponding replica set.

[0052] The data distribution method based on consistent hashing provided in this embodiment assigns a weight to each replica set by considering the storage performance differences of the replica sets, and divides the intervals for the replica sets according to the replica weights based on the consistent hashing method. The larger the replica weight, the larger the interval, and the object data is evenly distributed in the interval, so that the replica set with a larger weight stores more object data. It can balance the storage performance of the replica set and data loading, solve the problem of uneven utilization of read and write resources, improve the read and write performance of the object storage system, optimize the space load balancing effect, and improve the random access efficiency.

[0053] The data distribution strategy of this application can balance the write load according to the performance of the replica set, giving full play to the advantages of the distributed system in processing concurrent requests.

[0054] The consistent hashing data distribution method distributes object data in replica sets, and then distributes object data in multiple replica sets in storage nodes. In the current storage method, the primary node is preferentially selected to store object data. Specifically, the storage node with the largest available capacity in the replica set is selected as the primary node, without considering the performance factor of the storage unit. The replica set is provided by a single primary node, and a single balance condition will lead to an imbalance between the performance of the storage unit and data loading. The single read-write primary node leads to low utilization of read and write resources in the replica set, and multi-level complex data migration jointly restricts the overall performance of the system.

[0055] Therefore, in this embodiment, another data allocation method based on consistent hashing is provided, which can be applied to the above-mentioned object storage system. Figure 2 is a flowchart of another data distribution method based on consistent hashing according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 2 The process sequence shown is limited. Figure 2 As shown, the process includes the following steps:

[0056] Step S201, determining the replica weights of multiple replica sets using the replica set storage performance.

[0057] Specifically, the above step S201 includes:

[0058] Step S2011, obtaining the storage performance of the replica set.

[0059] The replica set storage performance includes the CPU status parameters, memory status parameters, disk status parameters, and network status parameters of each replica set.

[0060] Step S2012, determining the replica weights of the multiple replica sets using the replica set storage performance.

[0061] The storage performance indexes of multiple replica sets are used for modeling to determine the replica weight of each replica set.

[0062] Specifically, for the resource feature cost of the replica set in the small file storage scenario, the least squares method is used to determine the weight.

[0063] Step S202, divide the consistent hash ring into intervals based on the replica weights to obtain intervals of the replica set.

[0064] See also Figure 3 , Figure 3 It is a schematic diagram of an object data storage method based on a consistent hash ring according to an embodiment of the present invention.

[0065] The consistent hash ring is divided into intervals according to the proportion of the replica weights of multiple replica sets to obtain the intervals corresponding to each replica set. The larger the replica weight, the larger the interval, and the intervals of each replica set do not overlap. According to the intervals corresponding to the replica sets, the replica sets are mapped to the consistent hash ring using a hash function algorithm.

[0066] Construct a hash function algorithm and use the hash function algorithm to map each replica set into the corresponding interval.

[0067] like Figure 3 As shown, for k replica sets, the consistent hash ring is divided into k intervals based on the replica sets. In this implementation, the hash space range of the consistent hash is [0,1]. Specifically, the interval [0,1] is divided into k sub-intervals, and the replica sets are mapped in [0,1] using a hash function algorithm to obtain the hash value of each replica set, and the replica sets 1, 2, 3, ..., j, ..., k are mapped to the consistent hash ring. Among them, the hash space between replica set k and replica set 1 is the interval interval of replica set 1, and the hash space between replica set 1 and replica set 2 is the interval interval of replica set 2.

[0068] Step S203, calculating a hash value of the object data based on a hash function, and mapping the object data to a consistent hash ring.

[0069] Construct a hash function algorithm, map the object data to the interval of the hash space, obtain the hash value of each object data, and map the object data to the consistent hash ring.

[0070] Specifically, Figure 3 As shown, a hash function algorithm is used to evenly map each object data in [0,1], obtain a hash value of each object data, and map the object data to a consistent hash ring. The object data is evenly distributed on the consistent hash ring.

[0071] Step S204: searching the consistent hash ring for object data within the interval of the replica set, and storing the object data in the replica set.

[0072] like Figure 3 As shown, the object data mapped on the consistent hash ring is distributed in multiple intervals, and the object data in each interval is stored in the corresponding replica set.

[0073] Specifically, for each object data, the replica set is searched in a clockwise direction on the consistent hash ring, and the first replica set found is the target replica set of the object data, and the object data is stored in the corresponding replica set. And so on, all object data are stored in the corresponding replica set.

[0074] Generally speaking, the larger the interval between replica sets on the consistent hash ring, the more object data it stores.

[0075] Furthermore, for each object data in the replica set, the object data is stored in a storage node.

[0076] Step S205: determining node weights of multiple storage nodes according to the storage performance of the storage nodes.

[0077] Build a comprehensive performance evaluation model for each storage node in the replica set, obtain storage performance data for each storage node, and determine the node weight of each storage node based on the comprehensive performance evaluation model.

[0078] Specifically, the above step S205 includes:

[0079] Step S2051, obtaining the storage performance of the storage node.

[0080] The storage performance of the storage node includes the CPU status parameters, memory status parameters, disk status parameters, and network status parameters of each storage node.

[0081] Step S2052: determining node weights of multiple storage nodes according to the storage performance of the storage nodes.

[0082] The storage performance indexes of multiple storage nodes are used for modeling to determine the replica weight of each storage node.

[0083] Specifically, for the resource feature cost of the storage node in the small file storage scenario, the least squares method is used to determine the weight.

[0084] Step S206: for a replica set, all object data are divided according to the node weights of the storage nodes, and the object data are stored in the corresponding storage nodes.

[0085] All object data are divided according to the ratio of the node weights of multiple storage nodes to obtain the object data corresponding to each storage node. The greater the node weight of the storage node, the more corresponding object data.

[0086] For example, a replica set includes five storage nodes, storing a total of 60 object data. The node weights of the five storage nodes are 0.3, 0.5, 0.4, 0.7, and 0.1, respectively. Among them, the grounding weight of the first storage node is 3 / 20, the grounding weight of the second storage node is 1 / 4, the grounding weight of the third storage node is 1 / 5, the grounding weight of the fourth storage node is 7 / 20, and the grounding weight of the fifth storage node is 1 / 20. The object data is divided, and the first storage node object data is 60×3 / 20=9, the second storage node object data is 60×1 / 4=15, the third storage node object data is 60×1 / 5=12, the fourth storage node object data is 60×7 / 20=21, and the fifth storage node object data is 60×1 / 20=3.

[0087] The data distribution method based on consistent hashing provided in this embodiment gives weights to each replica set and storage node by considering the storage performance differences between replica sets and storage nodes, further stores data in the replica set according to the performance of the storage node, and through the two-level organization of the replica set and the storage node in the replica set when storing objects, it can meet the fault tolerance of object storage and can further balance the storage performance and data loading of the storage node. Based on the consistent hashing method, the replica set is divided into intervals according to the replica weights. The larger the replica weight, the larger the interval, and the object data is evenly distributed in the interval, so that the replica set with a larger weight stores more object data. It can balance the storage performance and data loading of the replica set, solve the problem of uneven utilization of read and write resources, improve the read and write performance of the object storage system, optimize the spatial load balancing effect, and improve the efficiency of random access.

[0088] Object storage systems have elastic expansion features and can trigger data migration by adding or deleting nodes. After the object storage system completes the task of object storage, when the scale of the object storage system changes, the object storage system needs to migrate a large amount of object data. Generally speaking, the standard hash method is the simplest data organization algorithm and can ensure balance, but when the system scale changes, the location of all data will change, resulting in greater performance jitter of the object storage system.

[0089] Among them, the scale change of the object storage system includes adding a new replica set or deleting a replica set.

[0090] In one implementation, after the object storage system completes the task of storing the object, when a new replica set is added, the object data is redistributed.

[0091] On this basis, in this embodiment, a method for allocating data when the scale of an object storage system changes is provided, which can be used in the above-mentioned object storage system. Figure 4 is a flow chart of a data allocation method under the change of object storage system scale according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment does not use Figure 4 The process sequence shown is limited. Figure 4 As shown, the process includes the following steps:

[0092] Step S401: when a new replica set is added, the replica weight of the replica set is re-determined based on the storage performance of the new replica set.

[0093] Specifically, the storage performance of the new replica set is obtained, and the replica weight of the new replica set is calculated using a comprehensive performance evaluation model.

[0094] Step S402: Determine a new interval of the old replica set on the consistent hash ring based on the replica weight.

[0095] The consistent hash ring is re-divided into intervals according to the proportion of the replica weights of multiple replica sets to obtain a new interval corresponding to each replica set.

[0096] Specifically, for k+1 replica sets, the consistent hash ring is divided into k+1 intervals based on the replica sets. In one implementation, the interval [0,1] is divided into k+1 sub-intervals. It can be understood that for the old replica sets 1,2,3,...,j,...,k, the original interval interval is reduced due to the addition of the new replica set k+1.

[0097] See also Figure 5 , Figure 5 The figure is a schematic diagram of object data migration according to an embodiment of the present invention.

[0098] like Figure 5 As shown, an old interval is used as an example for explanation. For replica set j, the old interval before adding the new replica set k+1 is interval 1, and after adding the new replica set, the new interval obtained by re-dividing is interval 2.

[0099] Step S403: determine a redundant interval between the new interval and the old interval.

[0100] The redundant part of the old interval with respect to the new interval is obtained as the redundant interval.

[0101] Specifically, Figure 5 As shown, the redundancy interval of replica set j is interval 3.

[0102] Step S404: Migrate the object data in the redundant interval from the old replica set to the new replica set.

[0103] For each replica set, a redundant interval is divided, and k redundant intervals are combined to form the interval interval of the new replica set k+1. The object data in each redundant interval is directly migrated from the old replica set to the new replica set, and other object data is not migrated.

[0104] Specifically, Figure 5 As shown, the object data within interval 2 continues to be stored in replica set j, and the object data within interval 3 is migrated to replica set k+1.

[0105] The data allocation method provided in this embodiment when the scale of the object storage system changes takes into account the storage scenario of the increase of replica sets, reallocates the interval interval according to the replica weights, does not move the object data in the new interval interval of the old replica set, and only migrates the object data in the redundant interval to the new replica set. The objects are adaptively migrated according to the increase in scale, and data migration does not occur between existing replica sets. The amount of data migration can be reduced, thereby reducing the system performance jitter caused by data migration, and optimizing the spatial load balancing effect.

[0106] In another implementation, after the object storage system completes the task of storing the object, the object data is redistributed when the replica set is deleted.

[0107] On this basis, another data allocation method under the change of object storage system scale is provided in this embodiment, which can be used for the above-mentioned object storage system. Figure 6 FIG. 1 is a flowchart of another method for allocating data when the object storage system scale changes according to an embodiment of the present invention. It should be noted that if there are substantially the same results, this embodiment is not limited to FIG. 1 . Figure 6 The process sequence shown is limited. Figure 6 As shown, the process includes the following steps:

[0108] Step S601: when deleting a replica set, re-determine the replica weight of the replica set based on the storage performance of the remaining replica sets.

[0109] Step S602: Determine a new interval of the remaining replica sets on the consistent hash ring based on the replica weights.

[0110] The consistent hash ring is re-divided into intervals according to the proportion of the replica weights of the multiple remaining replica sets to obtain a new interval corresponding to each replica set.

[0111] Specifically, for k-1 replica sets, the consistent hash ring is divided into k-1 intervals based on the replica sets. In one implementation, the interval [0,1] is divided into k-1 sub-intervals. It can be understood that for the remaining replica sets 1, 2, 3, ..., j, ..., k-1, the original interval increases due to the deletion of this set k.

[0112] See also Figure 7 , Figure 7 is another schematic diagram of object data migration according to an embodiment of the present invention.

[0113] like Figure 7 As shown, a remaining interval is used as an example for explanation. For replica set j, the old interval before deleting replica set k is interval 4, and after deleting replica set k, the new interval obtained by repartitioning is interval 5.

[0114] Step S603: determine an extension interval between the new interval and the old interval.

[0115] Get the extension of the new interval to the old interval as the extension interval.

[0116] Specifically, Figure 7 As shown, the expansion interval of replica set j is expansion interval 6.

[0117] Step S604: Migrate the object data in the deleted replica set to the remaining replica sets according to the expansion interval.

[0118] For each remaining replica set, an expansion interval is divided. Figure 7 As shown, the deletion replica set k is divided into its intervals according to the ratio of k-1 expansion intervals to obtain k-1 deletion intervals, and the object data in the deletion intervals are migrated to the corresponding remaining replica sets.

[0119] Specifically, Figure 7 As shown, the jth deletion interval is migrated to replica set j.

[0120] The data allocation method provided by this embodiment under the change of the scale of the object storage system takes into account the storage scenario of replica set deletion, reallocates the interval interval according to the replica weight, does not move the object data in the old interval interval of the old replica set, and only migrates the object data in the interval interval of the deleted replica set to each remaining replica set respectively, and adaptively migrates objects according to the scale. Data migration will not occur between existing replica sets, which can reduce the amount of data migration, thereby reducing the system performance jitter caused by data migration, and can optimize the spatial load balancing effect.

[0121] The data distribution method based on consistent hashing of the present application is applicable to an object storage system with a large number of small files, and can achieve balanced distribution of a large number of small files in the object storage system. As the number of concurrent reads increases, the advantages of the present application will become more and more obvious.

[0122] The present application can realize a weight-based balanced layout for the hierarchical organization of object data. Moreover, the balanced layout of the object storage system realizes the balanced layout of the read and write loads in the system, and can adapt to the large-scale random read and write access scenarios commonly seen in small file applications. It can give full play to the advantages of the parallelism of the object storage system and effectively improve the randomness of small files in the object storage system.

[0123] In this embodiment, a data distribution device based on consistent hashing is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] This embodiment provides a data distribution device based on consistent hashing. Figure 8 is a structural block diagram of a data distribution device based on consistent hashing according to an embodiment of the present invention. Figure 8 As shown, the data distribution device based on consistent hashing includes:

[0125] A weight calculation module 801 is used to determine the replica weights of the plurality of replica sets using the storage performance of the replica set;

[0126] An interval partitioning module 802 is used to partition the consistent hash ring based on the replica weight to obtain an interval of the replica set, wherein the larger the replica weight, the larger the interval;

[0127] A data mapping module 803, configured to calculate a hash value of the object data based on the hash function, and map the object data to the consistent hash ring;

[0128] The data storage module 804 is used to search the consistent hash ring for object data within the interval of the replica set, and store the object data in the replica set.

[0129] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0130] The data distribution device based on consistent hashing in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0131] The embodiment of the present invention also provides a computer device having the above Figure 8 A data distribution device based on consistent hashing is shown.

[0132] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0133] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0134] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0135] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0137] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0138] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0139] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0140] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0141] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data allocation method based on consistent hashing, applied to an object storage system, characterized in that: The object storage system includes multiple replica sets, and the method includes: Determining the replica weights of the plurality of replica sets using the replica set storage performance; Based on the replica weight, the consistent hash ring is divided into intervals to obtain the interval interval of the replica set, wherein the larger the replica weight, the larger the interval interval; Calculate a hash value of the object data based on a hash function, and map the object data to the consistent hash ring; The consistent hash ring is used to search for object data within the interval of the replica set, and the object data is stored in the replica set.

2. The data distribution method based on consistent hashing according to claim 1 is characterized in that: The replica set storage performance includes a CPU status parameter, a memory status parameter, a disk status parameter, and a network status parameter of each of the replica sets.

3. The data distribution method based on consistent hashing according to claim 1 is characterized in that: The interval intervals of the replica set obtained by dividing the consistent hash ring into intervals based on the replica weights include: Dividing the consistent hash ring into intervals according to the ratio of the replica weights of the multiple replica sets to obtain the interval interval corresponding to each replica set; The replica set is mapped to a corresponding interval using the hash function.

4. The data distribution method based on consistent hashing according to claim 1 is characterized in that: Each of the replica sets includes a plurality of storage nodes, and searching the consistent hash ring for object data within an interval of the replica set and mapping the object data to the replica set further includes: Determining node weights of a plurality of storage nodes according to storage performance of the storage nodes; For a replica set, all the object data are divided according to the node weights of the storage nodes, and the object data are stored in the corresponding storage nodes.

5. The data distribution method based on consistent hashing according to claim 1 is characterized in that: After searching the consistent hash ring for object data within the interval of the replica set and mapping the object data to the replica set, the method further includes: When a new replica set is added, the replica weight of the replica set is re-determined based on the storage performance of the new replica set; Determine a new interval of the old replica set on the consistent hash ring based on the replica weight; Determining a redundancy interval between the new interval and the old interval; The object data in the redundant interval is migrated from the old replica set to the new replica set.

6. The data distribution method based on consistent hashing according to claim 1 is characterized in that: After searching the consistent hash ring for object data within the interval of the replica set and mapping the object data to the replica set, the method further includes: When deleting a replica set, re-determining the replica weight of the replica set based on the storage performance of the remaining replica sets; Determine a new interval of the remaining replica sets on the consistent hash ring based on the replica weights; Determine an extension interval between the new interval and the old interval; Migrate the object data in the deleted replica set to the remaining replica sets according to the expansion interval.

7. A data distribution device based on consistent hashing, characterized in that: The device comprises: A weight calculation module, used to determine the replica weights of the plurality of replica sets using the storage performance of the replica set; An interval division module is used to divide the consistent hash ring into intervals based on the replica weight to obtain the interval interval of the replica set, wherein the larger the replica weight is, the larger the interval interval is; A data mapping module, used to calculate a hash value of object data based on a hash function, and map the object data to the consistent hash ring; The data storage module is used to search the consistent hash ring for object data within the interval of the replica set, and store the object data in the replica set.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the consistent hashing data allocation method described in any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the consistent hashing data allocation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, which are used to cause a computer to execute the consistent hashing data allocation method according to any one of claims 1 to 6.

Images