Data sharding method, device, electronic device, and storage medium

Abstract

The present application discloses a data sharding method, device, electronic device and storage medium. The method includes: obtaining multiple data to be stored on an initial hash ring; clustering each data to be stored to obtain a target clustering result of each data to be stored; for each target clustering result, calculating the mean of each eigenvector in the target clustering result; for each target clustering result, based on the mean and the preset weight of the target clustering result, determining the resource occupancy index of each data to be stored; based on the resource occupancy index of each data to be stored, updating the storage position of each data to be stored on the initial hash ring. The data sharding method of the present application can achieve balanced distribution of data to each data storage node, ensure that the system resources of each data storage node are relatively balanced, solve the problem of data skew caused by excessive data movement and single evaluation dimension in the data sharding process of traditional algorithms, thereby improving data sharding efficiency and system performance.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data sharding method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of the internet, operators are constantly generating massive amounts of data. This data is typically stored using a distributed storage approach, specifically by sharding the data and then storing it on various storage nodes. This approach to storing data on various storage nodes using data sharding typically employs hash tables (i.e., hashing) and consistent hash tables (i.e., consistent hashing).

[0003] Currently, data storage using a hashing method maps records to a slot in a table based on their key values. The hashing data sharding method calculates a hash value based on a characteristic (key) of the data and maps the hash value to the storage nodes in the system, thereby distributing data with different hash values ​​to different storage nodes. Using a consistent hashing method to store data maps the data to a connected hash ring based on its characteristic value, and also maps the storage nodes to this ring. For data, the first storage node found clockwise from the data's position on the ring is the data's storage node.

[0004] With the aforementioned hash-based data sharding storage method, when adding or removing a storage node, a large amount of data needs to be moved, resulting in an uneven distribution of the original data's characteristic values. This can lead to a large amount of data being concentrated on a single storage node, leading to an unbalanced load between storage nodes. With consistent hashing, when adding storage nodes, the pressure on only one existing storage node is shared. Similarly, if a storage node becomes unavailable, the pressure on that storage node is transferred entirely to the next storage node, resulting in data skew. This means that some storage nodes may store too much data, while others may store too little. This leads to an unbalanced load between storage nodes, impacting system performance. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a data sharding method, device, electronic device and storage medium to achieve adaptive and balanced distribution of data to each data storage node, ensure that the system resources of each data storage node are relatively balanced, and solve the problem of data skew caused by excessive data movement and a single evaluation dimension in the traditional algorithm during the data sharding process, thereby improving data sharding efficiency and system performance.

[0006] The technical solution of this application is as follows:

[0007] In a first aspect, a data sharding method is provided, the method comprising:

[0008] Obtain multiple data to be stored on the initial hash ring;

[0009] Clustering the data to be stored to obtain target clustering results for the data to be stored;

[0010] For each target clustering result, calculating the mean of each eigenvector in the target clustering result;

[0011] For each target clustering result, based on the mean and the preset weight of each eigenvector in the target clustering result, determining a resource occupancy index of each of the data to be stored;

[0012] Based on the resource occupation index of each of the data to be stored, the storage location of each of the data to be stored on the initial hash ring is updated.

[0013] In a second aspect, a data sharding device is provided, the device comprising:

[0014] An information acquisition module is used to acquire a plurality of data to be stored on the initial hash ring, and each feature vector corresponding to each of the data to be stored;

[0015] a target clustering result determination module, configured to cluster the data to be stored to obtain a target clustering result for the data to be stored;

[0016] A mean determination module, configured to calculate, for each target clustering result, the mean of each eigenvector in the target clustering result;

[0017] a resource occupancy index determination module, configured to determine, for each target clustering result, a resource occupancy index of each of the data to be stored based on the mean and a preset weight of each eigenvector in the target clustering result;

[0018] The data shard storage module is used to update the storage location of each data to be stored on the initial hash ring based on the resource occupation index of each data to be stored.

[0019] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data sharding method described in any embodiment of the present application.

[0020] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the data sharding method described in any embodiment of the present application are implemented.

[0021] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0022] The data sharding method provided by the embodiment of the present application obtains a target clustering result by clustering a plurality of data to be stored on the initial hash ring, and calculates the mean of each characteristic vector in each target clustering result for each target clustering result and the characteristic vector of each data to be stored in each target clustering result, and comprehensively evaluates the characteristic vector of each data to be stored, comprehensively weighs the system resources, and solves the problem of single evaluation dimension in the prior art. In addition, the present application updates the storage position of each data to be stored on the initial hash ring based on the resource occupancy index of each data to be stored. In this way, the data to be stored is adaptively and evenly distributed to each storage node according to its own resource occupancy index, ensuring that the system resources of each storage node are relatively balanced. By updating the storage position of each data to be stored on the initial hash ring, the problems of excessive data movement, data skew caused by a single evaluation dimension, and load imbalance between storage nodes in the traditional algorithm during data sharding are solved, thereby improving data sharding efficiency and system performance.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0025] Figure 1 This is a schematic diagram of the method of using hashing to perform data sharding provided by existing technology.

[0026] Figure 2 This is a schematic diagram of the migration of data to be stored when adding a storage node when using the hash method for data sharding provided by the existing technology;

[0027] Figure 3 This is a schematic diagram of the method of data sharding using consistent hashing provided by the existing technology;

[0028] Figure 4 This is a flow diagram of a data sharding method provided in an embodiment of the present application. Figure 1 ;

[0029] Figure 5This is a flow diagram of a data sharding method provided in an embodiment of the present application. Figure 2 ;

[0030] Figure 6 This is a schematic diagram of a process for clustering data to be stored provided in an embodiment of the present application;

[0031] Figure 7 Schematic diagram of a process for calculating resource occupancy indicators of each data to be stored provided in an embodiment of the present application;

[0032] Figure 8 This is a structural diagram of a data sharding device provided in an embodiment of the present application;

[0033] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make those of ordinary skill in the art better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0035] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples consistent with certain aspects of the present application as detailed in the appended claims.

[0036] To better understand the technical solution of this application, let's first introduce the background technology of this solution. The current storage of massive data basically uses a distributed storage method, specifically sharding the data and then storing the data on each storage node. The data sharding method is used to store data on each storage node, mostly using a hash table method (i.e., a hash method) and a consistent hash table method (i.e., a consistent hash method).

[0037] Data sharding is to divide the data set into independent and orthogonal data subsets according to certain rules, and then distribute the data subsets to different storage nodes. The quality of the sharding algorithm directly affects the performance of data storage.

[0038] The following describes how to implement data sharding using existing hashing methods and consistent hashing methods:

[0039] 1. Hash method

[0040] Hash tables are the most common data structures. They map records (or objects) to slots within the table based on their key values, facilitating fast access. Hash data sharding calculates a hash value based on a characteristic (key) of the data to be stored and maps this hash value to storage nodes in the system, distributing data with different hash values ​​to different storage nodes.

[0041] If you choose the id of the data to be stored as the key of the data shard, assuming that Node i Node is the storage node of i. num is the total number of storage nodes. The storage nodes for each data to be stored can be determined according to the following formula (1):

[0042] Node i =key%Node num

[0043] Assume there are 3 storage nodes, Node num =3, according to the above formula (1), the key of a certain data to be stored is modulo 3, when Node i The result of the calculation is 0, and the data to be stored is stored in the storage node Node0. i The calculation result is 1, and the data to be stored is stored in Node1. And so on, we can get the following Figure 1 Schematic diagram of the hash algorithm sharding results.

[0044] From the above calculation method, we can see that when data is sharded according to the hash method, the mapping relationship is very simple, and the amount of stored data that needs to be managed is very small. We only need to record the number of storage nodes and the hash method.

[0045] However, the disadvantages of the above hashing method are also very obvious: when adding or deleting a node, a large amount of data to be stored needs to be moved, and the migration cost is too high. For example, if a node Node3 is added here, according to the calculation method of the above formula (1), each data to be stored will be migrated accordingly, that is, the storage node of each data to be stored will change. The specific migration results are as follows: Figure 2In this way, when the characteristic values ​​of the data to be stored are unevenly distributed, a large amount of data to be stored may be concentrated on one storage node, causing an imbalance in the data to be stored on the storage node, that is, an imbalance in the load between the storage nodes.

[0046] 2. Consistent hashing method

[0047] Consistent hashing is to map the data to be stored onto a hash ring that is connected end to end according to the characteristic value, and also map the storage nodes onto this ring. For a certain data to be stored, starting from the position of the data to be stored on the ring, the first node found clockwise is the storage node of the data to be stored. Here we still take the above-mentioned data to be stored as an example. Assuming that the range of id is [0,1000], the positions of Node0, Node1, and Node2 on the ring are 100, 400, and 800 respectively. The data to be stored with id in [800,1000] and [0,100] are stored in Node0, the data to be stored with id in [100,400] are stored in Node1, and the data to be stored with id in [400,800] are stored in Node2. Then the hash ring diagram and the distribution of the data to be stored can be shown as follows: Figure 3 shown.

[0048] pass Figure 3 It can be seen that compared with the above hash method, the consistent hash method needs to maintain the additional data to be stored, including the position of the storage node on the ring, but the amount of this data is also very small.

[0049] When adding or deleting storage nodes, the affected data to be stored is relatively limited. For example, a node Node3 is added here, and its position on the ring is 600. Therefore, the range segment [400, 800] originally responsible for Node2 is now responsible for the range segment [400, 600], and the range segment [600, 800] is under the responsibility of Node3. Therefore, only records R2 (id: 759) and R3 (id: 607) need to be migrated from Node2 to Node3.

[0050] It is not difficult to find that the consistent hashing method will only affect the corresponding storage nodes on the hash ring when adding or deleting storage nodes, and no large-scale data migration will occur.

[0051] However, the consistent hashing algorithm has some shortcomings: First, when adding nodes, the consistent hashing method can only share the pressure of an existing storage node. Similarly, when one storage node becomes unavailable, the pressure of that storage node will be transferred to the next storage node, resulting in data skew. In other words, the data storage capacity of some storage nodes is too large, while the data storage capacity of some storage nodes is too small. This will cause performance bottlenecks in some storage nodes and affect the overall performance of the storage system. Second, the consistent hashing algorithm only allocates data based on the hash value of the data to be stored, without comprehensively considering other dimensions such as read and write frequency, usage frequency, and data size, which can also cause data skew.

[0052] In order to solve the problem of unbalanced load on each storage node caused by the above-mentioned hash method and consistent hash method when sharding data, the present application provides a data sharding method to solve the problem of data skew caused by excessive data movement and single evaluation dimension during the data sharding process in the prior art. The data sharding method provided by the present application can realize the adaptive and balanced distribution of the data to be stored to each data storage node, ensuring that the system resources of each data storage node are relatively balanced, thereby improving data sharding efficiency and system performance.

[0053] The data sharding method provided by this application is described in detail below.

[0054] In one example, in order to solve the problem in the prior art of data skew caused by excessive data movement and a single evaluation dimension during data sharding, the present application provides a data sharding method.

[0055] See also Figure 4 The data sharding method provided in the embodiment of the present application may specifically include the following steps:

[0056] S110: Acquire multiple data to be stored on the initial hash ring.

[0057] The initial hash ring can be as follows Figure 3 The hash ring in .

[0058] There are multiple data to be stored on the storage hash ring. These multiple data to be stored have been pre-stored on the initial hash ring. The specific pre-storage process will be described in detail in subsequent embodiments and will not be introduced in detail here.

[0059] S120: Cluster the data to be stored to obtain target clustering results for the data to be stored.

[0060] The target clustering result may be a clustering result of each data to be stored obtained after clustering each data to be stored.

[0061] After obtaining the data to be stored on the initial hash ring, each data to be stored can be clustered. Specifically, the k-mean algorithm can be improved and the improved k-mean algorithm can be used to cluster the data to be stored to obtain the target clustering result for each data to be stored. The specific use of the improved k-mean algorithm to cluster the data to be stored will be described in detail in subsequent embodiments and will not be detailed here.

[0062] S130 . For each target clustering result, calculate the mean of each eigenvector in the target clustering result.

[0063] The feature vector may be a vector used to represent each piece of data to be stored.

[0064] For each data to be stored in each target clustering result, each data to be stored has a preset number of feature vectors.

[0065] In one example, the feature vector may include, but is not limited to: query frequency, update frequency, size of data to be stored, number of fields of data to be stored, number of users querying data to be stored, and average response time of data to be stored.

[0066] In one example, the query frequency, update frequency, size of data to be stored, number of data fields to be stored, number of users querying the data to be stored, and average response time of the data to be stored can be calculated using a day as the time unit.

[0067] In an example, the query frequency may be the number of times the data to be stored is queried in a day.

[0068] In one example, the update frequency may be the number of times the data to be stored is updated in a day.

[0069] In an example, the size of the data to be stored may be the size of the disk occupied by the data to be stored.

[0070] In one example, the number of data fields to be stored may be the number of fields of the data to be stored.

[0071] In an example, the number of users who query the data to be stored may be the number of users who query the data to be stored in one day.

[0072] In an example, the average response time of the data to be stored may be an average response time of operating the data to be stored.

[0073] After obtaining the target clustering results, for each target clustering result, the mean of each eigenvector in the target clustering result may be calculated.

[0074] By obtaining multiple feature vectors of the data to be stored, the problem of data skew caused by a single evaluation dimension in the prior art, which is that only the feature values ​​of the primary key of the data to be stored are used to evaluate the range segments of the data to be stored, is solved. By using multiple feature vectors in this application, the range segments of the data to be stored are evaluated in multiple dimensions, making the evaluation of the range segments of the data to be stored more accurate.

[0075] S140 : For each target clustering result, determine the resource occupation index of each to-be-stored data based on the mean and the preset weight of each feature vector in the target clustering result.

[0076] The preset weight may be a preset weight of each feature vector in each target clustering result.

[0077] The resource occupation index may be the size of system resources occupied by each data to be stored, and specifically may be the size of node computing resources, storage resources, and network resources occupied by each data to be stored.

[0078] After obtaining the mean of each eigenvector of each target clustering result, the resource occupancy index corresponding to each eigenvector can be determined based on the mean and the preset weights of each eigenvector in each target clustering result. The resource occupancy index corresponding to each eigenvector obtained can be weighted calculated to obtain the resource occupancy index of each data to be stored in the target clustering result.

[0079] S150: Based on the resource occupation index of each data to be stored, update the storage location of each data to be stored on the initial hash ring.

[0080] After the resource occupation index of each data to be stored is obtained, the storage location of each data to be stored on the initial hash ring can be updated according to the resource occupation index of each data to be stored.

[0081] In one example, the specific update of the storage location of each data to be stored on the initial hash ring can be to make the sum of the resource occupancy index of each data to be stored on each storage node on the initial hash ring after the update not differ much, that is, based on the principle that each storage node has the same storage weight, the data to be stored is adaptively and evenly distributed to each storage node to ensure that the system resources of each storage node are relatively balanced.

[0082] Moreover, each data to be stored has been stored on the initial hash ring in advance according to the consistent hashing algorithm. When the storage location of each data to be stored on the initial hash ring is updated according to the resource occupancy index of each data to be stored, only the data to be stored that needs to be adjusted needs to be updated accordingly, without the need for a large amount of migration of the data to be stored. For example, Figure 3 For example, the data to be stored has been pre-stored on the initial hash ring, such as Figure 3As shown, after calculating the resource occupancy index of each data to be stored, if R0 and R5 need to be swapped to balance the load on each storage node, it is only necessary to swap the positions of R0 and R5. In this way, compared with not storing each data to be stored on the initial hash ring in advance, each data to be stored is directly stored on the hash ring according to the calculated resource occupancy index of each data to be stored, which reduces the amount of data movement and solves the problem of excessive data movement in the data sharding process in the prior art.

[0083] The technical solution of the present application is to cluster multiple data to be stored on the obtained initial hash ring to obtain a target clustering result, calculate the mean of each feature vector in each target clustering result for each target clustering result and the feature vector of each data to be stored in each target clustering result, comprehensively evaluate the feature vector of each data to be stored, comprehensively weigh the system resources, solve the problem of single evaluation dimension in the existing technology, and determine the resource occupancy index of each data to be stored based on the mean and the preset weight of each feature vector in the target clustering result for each target clustering result. Based on the resource occupancy index, the storage position of each data to be stored on the initial hash ring is updated. In this way, the initial consistent hashing algorithm is improved, and based on the principle that each storage node has the same storage weight, the data to be stored is adaptively and evenly distributed to each storage node, ensuring that the system resources of each storage node are relatively balanced. By updating the storage position of each data to be stored on the initial hash ring, the problems of excessive data movement, data tilt caused by a single evaluation dimension, and load imbalance between storage nodes in the traditional algorithm during data sharding are solved, thereby improving data sharding efficiency and system performance.

[0084] In one example, before obtaining each data to be stored on the initial hash ring, each data to be stored needs to be stored on the initial hash ring first. In order to store each data to be stored on the initial hash ring, as another implementation method of the present application, the present application provides another implementable method of the data sharding method. Please refer to the following examples for details.

[0085] See Figure 5 Another implementation of the data sharding method provided in this application specifically includes the following steps:

[0086] S210: Obtain the primary key of each data to be stored.

[0087] The primary key can be the primary key of the data to be stored. The specific format of the data to be stored is key:value format, where the primary key is the key of the data to be stored.

[0088] Before storing each data to be stored, the primary key of each data to be stored must be obtained first.

[0089] S220: Calculate a hash code value for the primary key to obtain a hash code value corresponding to each data to be stored.

[0090] After obtaining the primary key of each data to be stored, the hash code value hashcode is calculated for the primary key to obtain the hash code value corresponding to each data to be stored.

[0091] In one example, a hash code value is calculated for the primary key to obtain a hash code value corresponding to each data to be stored. The calculation can be performed according to the following formula to obtain a hash code value, which is an integer value.

[0092] s[0]*31 n-1 +s[1]*31 n-2 +…+s[n-1];

[0093] Among them, s[i] is the primary key of the data to be stored, i is 1, 2, ..., n-1; the storage nodes on the initial hash ring are 2 to the power of 32, and the 31 here corresponds to 32, because from 1 to 31 n-1 That is 32 numbers.

[0094] Correspondingly, the implementation code for calculating the hash code value of the primary key and obtaining the hash code value corresponding to each data to be stored can be as follows:

[0095]

[0096]

[0097] S230 , performing hash calculation on the hash code value to obtain a hash value corresponding to each data to be stored.

[0098] After obtaining the hash code value corresponding to each data to be stored, the hash code value is hashed to obtain the hash value corresponding to each data to be stored.

[0099] Correspondingly, the implementation code for performing hash calculation on the hash code value to obtain the hash value corresponding to each data to be stored can be as follows:

[0100]

[0101] S240 . Allocate each data to be stored to a corresponding storage node on the initial hash ring based on the hash value.

[0102] According to the hash values ​​corresponding to the data to be stored obtained above, the data to be stored can be allocated to the corresponding storage nodes on the initial hash ring.

[0103] In one example, the specific allocation of each to-be-stored data to the storage node corresponding to the initial hash ring can be based on the method of storing the to-be-stored data on the hash ring in the consistent hash algorithm in the prior art, which is not described in detail here.

[0104] Through the above consistent hash algorithm, the primary key of any data to be stored is mapped into a unique 0-2 32 -1, allocate the data to be stored to the corresponding data slot (storage node), such as the data to be stored with hash value N is mapped to the data slot N.

[0105] S250: Acquire multiple data to be stored on the initial hash ring.

[0106] S260: Cluster the data to be stored to obtain target clustering results for the data to be stored.

[0107] S270: For each target clustering result, calculate the mean of each eigenvector in the target clustering result.

[0108] S280 : For each target clustering result, determine the resource occupation index of each data to be stored based on the mean and the preset weight of each feature vector in the target clustering result.

[0109] S290: Based on the resource occupation index of each data to be stored, update the storage location of each data to be stored on the initial hash ring.

[0110] In one example, the above steps S250-S290 are the same as steps S110-S150 in the above embodiment, and for the sake of brevity, they are not described in detail here.

[0111] The technical solution of the embodiment of the present application is to obtain a hash code value corresponding to each data to be stored by performing hash code value calculation on the primary key of each data to be stored, and then perform hash calculation on the hash code value to obtain a hash value corresponding to each data to be stored. Based on each hash value, each data to be stored is allocated to each storage node on the initial hash ring, so that the storage position of each data to be stored on the initial hash ring can be updated subsequently according to the resource occupancy index of each data to be stored.

[0112] In one example, in order to cluster the data to be stored on the initial hash ring, the existing k-mean algorithm may be improved, and the improved k-mean algorithm may be used to cluster the data to be stored.

[0113] In order to achieve clustering of the data to be stored on the initial hash ring, as another feasible method of the present application, the embodiment of the present application also provides another feasible implementation method of the data sharding method, which can be a method of clustering the data to be stored on the initial hash ring. For details, please refer to the following embodiments.

[0114] Step S260 may specifically include the following steps:

[0115] S2601: Determine a plurality of cluster nodes based on each data to be stored, the quantity of each data to be stored, and a preset number of clusters.

[0116] The preset number of clusters may be the number of clustering results that is preset, for example, the number of clusters into which all the data to be stored are clustered.

[0117] Clustering nodes can be nodes of each cluster. For example, if there are 6 data to be stored and these 6 data to be stored are clustered into 2 categories, there will be 2 clustering nodes. These 6 data to be stored can be clustered based on these 2 clustering nodes. Specifically, the distance between each data to be stored and the 2 clustering nodes can be calculated. The data to be stored is assigned to the category of the clustering node to which the data is closest.

[0118] According to each data to be stored, the quantity of each data to be stored and the preset number of clusters, multiple cluster nodes may be determined. Specifically, each cluster node may be determined according to the following method:

[0119] a. Assume x i ={a1,a2,a3...a6} is an item to be classified, where x i is a data to be stored, and each a is x i A feature vector of , specifically: a1 = query frequency within 1 day; a2 = update frequency within 1 day; a3 = size of data to be stored; a4 = number of data fields to be stored; a5 = number of users who query the data to be stored within 1 day; a6 = average response time for data to be stored within 1 day.

[0120] b. This application plans to divide the data set to be stored into 5 clusters, that is, the number of preset clusters is 5. Assume that the cluster node set is C = {C1, C2, C3, C4, C5}, where each C represents a cluster node. To address the shortcomings of the k-means clustering algorithm, which may cause inaccurate clustering results if the cluster nodes are selected too densely, an optimized cluster node selection method is adopted. The specific selection method is as follows:

[0121] Determination of the first clustering node: Select the center point of all the data to be stored as the initial clustering node C1, and calculate the initial clustering node C1 based on the following formula:

[0122]

[0123] Among them, x all is the set of all data to be stored, S all is the total amount of data to be stored,

[0124] Selection of the second clustering node: all The data to be stored that is farthest from C1 is selected as C2 to ensure that the selection of each clustering node is relatively dispersed. The specific calculation method is as follows:

[0125]

[0126] Among them, X i For any data to be stored in the set of all data to be stored, is x all The distance between the data to be stored and C1.

[0127] That is, traverse x all Among the data to be stored, the data to be stored that is farthest from C1 is selected as C2.

[0128] Similarly, according to the calculation formula of C2 above, the third clustering node can be calculated, specifically at x all Select the data to be stored that is farthest from C1 and C2.

[0129] Similarly, calculate the fourth cluster node, specifically in x all Select the data to be stored that is farthest from C1, C2 and C3.

[0130] Similarly, calculate the fifth cluster node, specifically in x all Select the data to be stored that is farthest from C1, C2, C3 and C4.

[0131] According to the above calculation method, each cluster node can be calculated.

[0132] S2602: Calculate the first Euclidean distance between each data to be stored and each cluster node.

[0133] The first Euclidean distance may be a Euclidean distance between each data to be stored and each clustering node.

[0134] After calculating each clustering node, the first Euclidean distance between each data to be stored and each clustering node may be calculated, so that each data to be stored can be clustered based on the first Euclidean clustering.

[0135] S2603: Put each data to be stored into the cluster with the smallest first Euclidean distance corresponding to the data to be stored, to obtain an initial clustering result of each data to be stored.

[0136] The initial clustering result may be a clustering result of each data to be stored obtained by clustering each data to be stored based on the first Euclidean distance.

[0137] After obtaining the first Euclidean distance between each data to be stored and each cluster node, each data to be stored can be placed in the cluster with the smallest first Euclidean distance corresponding to the data to be stored. That is, for each data to be stored, after calculating the first Euclidean distance between the data to be stored and each cluster node, the data to be stored is placed in the cluster corresponding to the cluster node with the smallest first Euclidean distance between the data to be stored and each cluster node.

[0138] In an example, specifically, the first Euclidean distances of the data A to be stored to each cluster node are 1, 2, 3, 4 and 5 respectively. Among these 5 first Euclidean distances, the first Euclidean distance of the data A to be stored to the first cluster node is the smallest (the first Euclidean distance is 1), then the data to be stored is placed in the cluster corresponding to the first cluster node (that is, the cluster node whose first Euclidean distance is 1).

[0139] In this way, according to the above method, each data to be stored can be clustered to obtain an initial clustering result of each data to be stored.

[0140] Specifically, the first Euclidean distance between each data to be stored and each cluster node can be calculated based on the following formula:

[0141]

[0142] Among them, X i For any data to be stored, C k are the cluster nodes, where k is 1, 2, 3, 4, or 5; It is the first Euclidean distance from a certain data to be stored to a certain cluster node.

[0143] According to the above formula, the first Euclidean distance of each data to be stored to the cluster center C = {C1, C2, C3, C4, C5} can be calculated respectively, which are disX i C1, disX i C2, disX i C3, disX i C4, disX i C5, according to the selected disX i C k Based on the principle of minimization, each data to be stored is divided into the cluster with the smallest Euclidean distance from the cluster center, and finally five clustering results are formed, that is, the five initial clustering results are as follows: S = {S1, S2, S3, S4, S5}, where S iis the initial clustering result.

[0144] After obtaining the initial clustering result, the following steps are repeatedly performed based on the initial clustering result until the cluster to which each data to be stored belongs does not change, and the target clustering result of each data to be stored is obtained:

[0145] S2604: For each initial clustering result, calculate the second Euclidean distance between each data to be stored in the initial clustering result and the center point of each initial clustering result, and put each data to be stored into the cluster with the smallest second Euclidean distance corresponding to the data to be stored.

[0146] The second Euclidean distance is the Euclidean distance between each to-be-stored data point in each initial clustering result and the center point of each initial clustering result, calculated for each initial clustering result. For example, for each of the five initial clustering results, such as initial clustering result S1, the center points of the five initial clustering results are first calculated. Then, the Euclidean distance between each to-be-stored data point in S1 and the center point of each initial clustering result is calculated. This Euclidean distance is the second Euclidean distance.

[0147] After obtaining the initial clustering results, for each initial clustering result, the second Euclidean distance between each data to be stored in the initial clustering result and the center point of each initial clustering result is calculated, and based on the principle of minimum second Euclidean distance, each data to be stored is placed in the cluster with the minimum second Euclidean distance corresponding to the data to be stored.

[0148] In one example, after obtaining the initial clustering results, new center points are selected from each initial clustering result to re-cluster each data to be stored. This is to avoid the situation where the clustering is not accurate enough.

[0149] In the embodiment of the present application, the calculation method of the second Euclidean distance is the same as the calculation formula of the first Euclidean distance mentioned above. For the sake of brevity, it will not be repeated here.

[0150] After each data item to be stored is placed into the cluster corresponding to the data item to be stored with the smallest second Euclidean distance based on the second Euclidean distance, the center points of each new clustering result are obtained. The Euclidean distance from each data item to be stored to the center point of each new clustering result is then calculated. The data item to be stored is then clustered based on the principle of minimizing the Euclidean distance. In other words, step S2603 is repeated until the data item to be stored in each cluster remains unchanged, at which point clustering is complete.

[0151] The technical solution of the embodiment of the present application is to determine multiple clustering nodes according to each data to be stored, the number of each data to be stored and the preset number of clusters; then calculate the first Euclidean distance of each data to be stored to each clustering node; based on the principle of minimizing the first Euclidean distance, obtain the initial clustering result of each data to be stored; repeatedly perform the steps of calculating the second Euclidean distance of each data to be stored in the initial clustering result to the center point of each initial clustering result for each initial clustering result, and placing each data to be stored into the cluster with the smallest second Euclidean distance corresponding to the data to be stored, until the cluster to which each data to be stored belongs does not change, and obtain the target clustering result of each data to be stored. In this way, an improved k-means clustering algorithm is used to improve the method of selecting each cluster node, so that the clustering algorithm is more accurate. According to the calculation result of the eigenvector, all the data to be stored are clustered to obtain an accurate target clustering result, so that the mean of the eigenvector of each data to be stored in each target cluster can be calculated based on the accurate target clustering result.

[0152] The above embodiment introduces in detail the method of clustering the data to be stored. In order to enable those skilled in the art to better understand how to obtain the target clustering results based on the initial clustering results, as another feasible embodiment of the present application, the present application also provides a data sharding method, which can be specifically how to obtain the target clustering results based on the initial clustering results. Please refer to the following embodiment for details.

[0153] Step S2604 may specifically include the following steps:

[0154] S26041. For each initial clustering result, based on the feature vector corresponding to each data to be stored in the initial clustering result and the number of each data to be stored in the initial clustering result, determine the center point of the initial clustering result.

[0155] After obtaining the initial clustering results, the center point of the initial clustering results can be determined according to the feature vectors corresponding to the data to be stored in the initial clustering results and the number of data to be stored in the initial clustering results. Specifically, the center point of each initial clustering result can be determined by referring to the following method:

[0156] For the first initial clustering result, assume that S1={x1,x2,...x k}, where S1 is the first initial clustering result, x i For each data to be stored in the initial clustering result S1, the calculation formula for the center point of the initial clustering result S1 is as follows:

[0157]

[0158] Among them, x iIt is in vector form, and specifically may be composed of the feature vectors corresponding to the data to be stored, and k is the number of data to be stored in the initial clustering result S1.

[0159] According to the above formula, the center point of the first initial clustering result S1 can be calculated. Similarly, based on the same calculation method as the above formula, the center points of the other four initial clustering results can be calculated, namely C 2-new 、C 3-new 、C 4-new and C 5-new .

[0160] S26042. Calculate the second Euclidean distance between each data to be stored and the center point of each initial clustering result.

[0161] After obtaining the center point of each initial clustering result, the second Euclidean distance between each data to be stored and the center point of each initial clustering result can be calculated. The specific calculation method of the second Euclidean distance is the same as the calculation method of the first Euclidean distance in the above embodiment, and will not be repeated here.

[0162] S26043. Place each data to be stored into the cluster corresponding to the data to be stored and having the smallest second Euclidean distance.

[0163] After obtaining the second Euclidean distance between each data to be stored and the center point of each initial clustering result, each data to be stored can be placed in the cluster with the smallest second Euclidean distance corresponding to the data to be stored based on the principle of minimum second Euclidean distance, so that each data to be stored is re-clustered.

[0164] Repeat the above steps S26041-S26043 until the cluster to which each data to be stored belongs does not change, then determine that the clustering is completed and obtain the target clustering result.

[0165] The technical solution of the embodiment of the present application is to repeatedly execute the above steps S26041-S26043 until the cluster to which each data to be stored belongs does not change, and then determine that the clustering is completed. In this way, an accurate target clustering result can be obtained, so that the mean of the characteristic vector of each data to be stored in each target cluster can be calculated based on the accurate target clustering result.

[0166] In one example, before clustering the data to be stored, it is first necessary to obtain the feature vectors of each data to be stored. After obtaining the feature vectors of each data to be stored, there may be a situation where the feature attributes of the data to be stored represented by two feature vectors are the same. At this time, it is necessary to filter the feature vectors to filter out the feature vectors with the same feature attributes to reduce the amount of calculation, improve calculation efficiency, and save hardware computing resources.

[0167] In order to screen each feature vector to filter out feature vectors with the same characteristic attributes, thereby reducing the amount of calculation, improving calculation efficiency, and saving hardware computing resources, the embodiment of the present application also provides another data sharding method, which can be specifically the screening of each feature vector. For details, please refer to the following embodiment.

[0168] See also Figure 6 The specific steps of using the improved k-mean algorithm to cluster the data to be stored are as follows:

[0169] 1. Selection of eigenvectors

[0170] In an embodiment of the present application, for each data to be stored, a feature vector of each data to be stored is selected in units of days. Specifically, the six feature vectors selected may include the query frequency, update frequency, size of data to be stored, number of data fields to be stored, number of users querying the data to be stored, and average response time of the data to be stored of each data to be stored within one day.

[0171] It should be noted that the six feature vectors selected above are only examples of the embodiments of the present application. Those skilled in the art should know that the feature vectors in the embodiments of the present application are not limited to the six feature vectors mentioned above. Any vector that can represent the characteristic attributes of the data to be stored can be used as the feature vector of the embodiments of the present application, that is, any vector that can represent the characteristic attributes of the data to be stored is within the protection scope of the embodiments of the present application.

[0172] 2. Calculation of eigenvectors

[0173] After selecting each eigenvector, statistical calculations may be performed on each eigenvector to obtain the value of each eigenvector of each data to be stored.

[0174] Specifically, for example, the values ​​of the eigenvectors in Table 1 below can be obtained:

[0175] Table 1: Final sample example

[0176]

[0177] It should be noted that the values ​​of the eigenvectors in Table 1 above are only used as an example and do not mean that the eigenvectors can only have this value. The values ​​of the eigenvectors are the values ​​of the eigenvectors within a day obtained based on statistics in actual applications.

[0178] 3. Feature vector screening

[0179] After obtaining the values ​​of each eigenvector, there may be two eigenvectors representing the same characteristic attributes, so the eigenvectors need to be screened. The specific screening process is as follows:

[0180] For the sample set in Table 1 above, any two eigenvectors are subjected to correlation analysis, where the correlation coefficient formula between the two eigenvectors is as follows:

[0181]

[0182] Among them, ρ XY Represents the similarity of two eigenvectors; X, Y represent two eigenvectors; D(X) represents the variance of the value of the eigenvector X, D(Y) represents the variance of the value of the eigenvector Y, and Cov(X,Y) represents the covariance matrix of the values ​​of the two eigenvectors X and Y.

[0183] In the above formula, ρ XY The larger it is, the more similar the two eigenvectors X and Y are.

[0184] According to the above formula, the correlation between any two feature vectors among query frequency, update frequency, data size, number of data fields to be stored, number of users querying data to be stored and average response time of data to be stored is calculated, and the correlation coefficient ρ is used. XY Only one eigenvector is selected that is greater than or equal to a certain threshold (for example, the threshold can be 0.9), ρ XY The larger the value, the stronger the correlation between the eigenvectors. The principle of eigenvector selection is to find vectors with little correlation to form the final sample set.

[0185] 4. Use the improved k-means clustering algorithm to train the model

[0186] After filtering each eigenvector, the filtered eigenvectors can be used to train a model based on an improved k-means clustering algorithm, and the model can be used to obtain target clustering results for each data to be stored.

[0187] (1) Using the feature vectors filtered in step 3, calculate the query frequency, update frequency, data size, number of fields in the data to be stored, number of users querying the data to be stored, and average response time of the data to be stored within one day for each data to be stored, as the sample set for this model training.

[0188] (2) Sample set division: the sample set is divided into training set and test set in a ratio of 8:2.

[0189] (3) An improved k-menas clustering algorithm is applied to the data to be stored in the training set. The specific steps of the algorithm are as described in S1201-S1204 in the above embodiment. Based on S1201-S1204 in the above embodiment, a target clustering result is obtained. Specifically, for example, the target clustering result shown in Table 2 below can be obtained:

[0190] Table 2: Example of clustered samples

[0191]

[0192] The technical solution of the embodiment of the present application reduces the amount of calculation, improves the calculation efficiency, and saves hardware computing resources by screening the feature vectors to filter out feature vectors with the same feature attributes.

[0193] In the above embodiment, each eigenvector is filtered, and based on the filtered eigenvectors, the data to be stored is clustered using an improved k-mean algorithm to obtain a target clustering result. In order to ensure that the resources occupied by the data to be stored stored on each storage node (resource occupation index) are consistent, after obtaining the target clustering result, it is necessary to calculate the resources occupied by each data to be stored in the target clustering result. For details, please refer to the following embodiment.

[0194] In order to make the resources occupied by the data to be stored on each storage node consistent, it is necessary to calculate the resources occupied by each data to be stored in the target clustering result. This application also provides another implementable method of the data sharding method, specifically calculating the resources occupied by each data to be stored in the target clustering result.

[0195] See also Figure 7 , the specific steps for calculating the resources occupied by each data to be stored in the target clustering result are as follows:

[0196] 1. Calculate the mean of each eigenvector of each target clustering result.

[0197] 2. Calculate the ranking of the mean of each eigenvector.

[0198] 3. Based on the weight and mean ranking of each eigenvector in the preset target clustering result, calculate the resource occupancy index of each data to be stored.

[0199] 4. Based on the resource usage indicators of each data to be stored, adaptive balanced sharding is performed.

[0200] Using the above method, the resource occupancy index of each data to be stored can be obtained. Based on the resource occupancy index of each data to be stored, the storage position of each data to be stored on the initial hash ring can be updated so that the total amount of resources occupied by each data to be stored on each storage node on the initial hash ring is consistent, thereby achieving the effect of load balancing on each storage node.

[0201] In order to introduce in detail the calculation of the mean of each eigenvector in the target clustering result, the present application also provides another feasible implementation method of the data sharding method, which can be seen in the following examples for details.

[0202] In one example, the mean of each feature vector in the target clustering result may be calculated in the following manner, that is, step S270 may include the following steps:

[0203] S2701 : For each current target clustering result, based on the eigenvector values ​​corresponding to the eigenvectors of each data to be stored in the current target clustering result, determine the mean of each eigenvector in the current target clustering result.

[0204] In an example, the current target clustering result may be the target clustering result to be calculated currently. For example, there are 5 target clustering results in total, and the first clustering result is to be calculated currently. Then, the first clustering result is the current target clustering result.

[0205] Assume that for the current target clustering result S1, S1 has multiple data to be stored, that is, S1={x1,x2,...x k}, each data to be stored x i It has 6 eigenvectors, namely x i ={a1,a2,a3...a6}, each data to be stored x i They are all represented by the value of their corresponding eigenvectors, that is, each data x to be stored i is a multidimensional vector. The mean value of the eigenvector a1 in S1 is calculated as follows:

[0206]

[0207] Here, k is the number of data to be stored in S1.

[0208] Similarly, we can calculate S1a2, S1a3, S1a4, S1a5, S1a6; S2a1, S2a2, S2a3, S2a4, S2a5, S2a6; ...; S5a1, S5a2, S5a3, S5a4, S5a5, S5a6. In this way, we can obtain the mean of each eigenvector in each target clustering result.

[0209] By obtaining multiple feature vectors of the data to be stored, the problem of data skew caused by a single evaluation dimension in the prior art, which is that only the feature values ​​of the primary key of the data to be stored are used to evaluate the range segments of the data to be stored, is solved. By using multiple feature vectors in this application, the range segments of the data to be stored are evaluated in multiple dimensions, making the evaluation of the range segments of the data to be stored more accurate.

[0210] In order to introduce in detail the calculation of the resource occupancy index of each data to be stored, the present application also provides another possible implementation method of the data sharding method, and the details can be found in the following examples.

[0211] In one example, the resource occupancy index of each to-be-stored data may be calculated in the following manner, that is, step S280 may include the following steps:

[0212] S2801. Based on the mean of each eigenvector in each target clustering result, sort by the mean to obtain a ranking result of each target clustering result.

[0213] After obtaining the mean of each eigenvector in each target clustering result, the means are sorted according to the size of the mean to obtain the sorting result of each target clustering result.

[0214] Specifically, for each target clustering result, the eigenvectors {a1, a2, a3...a6} may be sorted according to their mean values ​​to obtain the ranking of each eigenvector in each target clustering result, for example, Table 3 may be obtained.

[0215] Table 3: Ranking of each eigenvector in each target clustering result

[0216]

[0217] The ranking of each eigenvector in each target clustering result has the following meanings in this application:

[0218] (1) The higher the eigenvector score, the higher the resource demand of the eigenvector;

[0219] (2) The query frequency is high and the demand for computing resources is large, so the ranking score is high (i.e., at the front);

[0220] (3) The update frequency is high and the demand for computing resources is large, so the score is high;

[0221] (4) The larger the data size, the greater the demand for storage resources, so the score is higher;

[0222] (5) The number of data fields is large, which requires a lot of storage resources, so the score is high;

[0223] (6) The number of users querying the data is large, indicating that the data is high-frequency data, so the score is high;

[0224] (7) The average data response time is long, indicating that the data has a high latency, so resources are required to generate it, resulting in a high score.

[0225] Based on the above principles, the ranking of each eigenvector can be finally shown in Table 4 below:

[0226] Table 4: Final ranking of each eigenvector

[0227]

[0228] S2802: For each current target clustering result, based on the ranking result of each eigenvector in the current target clustering result and the preset weight of each eigenvector in the current target clustering result, determine the resource occupation index of each to-be-stored data in the current target clustering result.

[0229] The current target clustering result may be the target clustering result to be currently calculated.

[0230] The preset weight may be a pre-set weight of each feature vector in each target clustering result. For each target clustering result, the preset weight of each feature vector may be as shown in Table 5 below:

[0231] Table 5: Preset weight evaluation of feature vectors

[0232]

[0233]

[0234] It should be noted that the preset weights of the eigenvectors listed in Table 5 above are only examples of the embodiments of the present application, and do not mean that the preset weights of the eigenvectors in the present application are the preset weights in Table 5 above. In the embodiments of the present application, the preset weights of the eigenvectors can be set according to user needs and are not limited here.

[0235] After obtaining the ranking of each eigenvector in each target clustering result, for each current target clustering result, based on the ranking results of each eigenvector in the current target clustering result and the preset weights of each eigenvector in the current target clustering result, the resource occupancy index of each data to be stored in the current target clustering result is determined. Specifically, the weights of each eigenvector in the current target clustering result can be obtained based on the ranking results of each eigenvector in the current target clustering result and the preset weights of each eigenvector in the current target clustering result, and the weighted calculation of each eigenvector in the current target clustering result can be performed to obtain the resource occupancy index of each data to be stored in the current target clustering result.

[0236] It should be noted that, because the resource occupancy index of each data to be stored in the current target clustering result is calculated based on the ranking results of each eigenvector in the current target clustering result and the preset weight of each eigenvector in the current target clustering result, the weight of each eigenvector in the current target clustering result is obtained. Each data to be stored in the current target clustering result has the same eigenvector. After the weight of each eigenvector in the current target clustering result is calculated, the weight of each eigenvector in the current target clustering result is weightedly calculated to obtain the weight of the current target clustering result, that is, the resource occupancy index of each data to be stored in the current target clustering result. Therefore, for a target clustering result, the resource occupancy index of each data to be stored in the target clustering result is the same.

[0237] In an example, the resource usage indicator calculation formula for each to-be-stored data in a target clustering result is as follows:

[0238] Resource occupancy index = ∑ the preset weight of each feature vector in the target clustering result * mean ranking.

[0239] The technical solution of the embodiment of the present application proposes a new method for calculating the resource occupancy index of the data to be stored. Specifically, the ranking results of each target clustering result are obtained by sorting according to the mean of each eigenvector in each target clustering result. For each current target clustering result, the resource occupancy index of each data to be stored in the current target clustering result is determined based on the ranking results of each eigenvector in the current target clustering result and the preset weight of each eigenvector in the current target clustering result. In this way, by comprehensively weighing the resource occupancy index of the data to be stored, different categories of data to be stored have different weights. According to the preset weight of the eigenvector and the ranking of each eigenvector in the target clustering result, the resource occupancy index of each data to be stored is calculated, so that the storage position of each data to be stored on the initial hash ring can be updated based on the resource occupancy index to ensure load balancing on each storage node on the initial hash ring.

[0240] In the above embodiment, the resource occupancy index of each data to be stored is obtained through calculation. In order to balance the load on each storage node on the initial hash ring, it is necessary to update the storage position of each data to be stored on the initial hash ring based on the resource occupancy index of each data to be stored.

[0241] In one example, after obtaining the resource occupancy index of each data to be stored, in order to balance the load on each storage node on the initial hash ring, the embodiment of the present application also provides another feasible implementation method of the data sharding method, which can be seen in the following embodiment for details.

[0242] In order to balance the load on each storage node in the initial hash ring, step S290 may specifically include the following steps:

[0243] S2901. Obtain each storage node on the initial hash ring and the storage weight corresponding to each storage node.

[0244] The storage weight may be the weight of the load on each storage node on the current initial hash ring, and specifically may be the sum of the resource occupancy indicators of the load on each storage node on the current initial hash ring.

[0245] After obtaining the resource occupancy index of each data to be stored, each storage node on the initial hash ring and the storage weight corresponding to each storage node can be obtained.

[0246] S2902. Based on the resource occupancy index and storage weight of each data to be stored, update the storage location of each data to be stored on the initial hash ring so that the difference in storage weight between each storage node is less than or equal to a preset difference threshold.

[0247] The preset difference threshold may be a preset threshold of the difference in storage weights between storage nodes.

[0248] After obtaining the resource occupancy index of each data to be stored, each storage node on the initial hash ring and the storage weight corresponding to each storage node, the storage position of each data to be stored on the initial hash ring can be updated based on the resource occupancy index of each data to be stored, each storage node on the initial hash ring and the storage weight corresponding to each storage node, so that the difference in storage weights between each storage node is less than or equal to the preset difference threshold. In this way, based on the principle that each storage node has the same storage weight, the data to be stored is adaptively and evenly distributed to each storage node, which can make the system resources of each storage node relatively balanced. By updating the storage position of each data to be stored on the initial hash ring, the problems of excessive data movement in the data sharding process, data skew caused by a single evaluation dimension, and load imbalance between storage nodes in traditional algorithms are solved, thereby improving data sharding efficiency and system performance.

[0249] The technical solution of the embodiment of the present application obtains each storage node on the initial hash ring and the storage weight corresponding to each storage node, and updates the storage position of each data to be stored on the initial hash ring based on the resource occupancy index of each data to be stored and the storage weight, so that the difference in storage weights between each storage node is less than or equal to a preset difference threshold. In this way, based on the principle that each storage node has the same storage weight, the data to be stored is adaptively and evenly distributed to each storage node, so that the system resources of each storage node can be relatively balanced. By updating the storage position of each data to be stored on the initial hash ring, the problems of excessive data movement, data skew caused by a single evaluation dimension, and load imbalance between storage nodes in the traditional algorithm during data sharding are solved, thereby improving data sharding efficiency and system performance.

[0250] Based on the data sharding method provided in the above embodiment, the present application also provides a specific implementation of the data sharding device, please refer to the following embodiments.

[0251] See first Figure 8 , the data sharding device provided in the embodiment of the present application includes the following modules:

[0252] The information acquisition module 410 is used to obtain a plurality of data to be stored on the initial hash ring, and each feature vector corresponding to each of the data to be stored;

[0253] A target clustering result determination module 420 is configured to cluster the data to be stored to obtain a target clustering result for each data to be stored;

[0254] A mean determination module 430 is configured to calculate, for each target clustering result, a mean of each eigenvector in the target clustering result;

[0255] The resource occupancy index determination module 440 is configured to determine, for each target clustering result, a resource occupancy index of each of the to-be-stored data based on the mean and a preset weight of each eigenvector in the target clustering result;

[0256] The data shard storage module 450 is configured to update the storage location of each of the data to be stored on the initial hash ring based on the resource occupancy indicator of each of the data to be stored.

[0257] The technical solution of the embodiment of the present application is to cluster multiple data to be stored on the initial hash ring obtained by the information acquisition module through the target clustering result determination module to obtain the target clustering result, and based on the mean determination module, for each target clustering result and the feature vector of each data to be stored in each target clustering result, calculate the mean of each feature vector in each target clustering result, comprehensively evaluate the feature vector of each data to be stored, comprehensively weigh the system resources, and solve the problem of single evaluation dimension in the prior art, based on the resource occupancy index determination module for each target clustering result, based on the mean and the preset weight of each feature vector in the target clustering result, determine the resource occupancy index of each data to be stored, and based on the data sharding storage module, update the storage position of each data to be stored on the initial hash ring based on the resource occupancy index. In this way, the initial consistent hashing algorithm is improved. Based on the principle that each storage node has the same storage weight, the data to be stored is adaptively and evenly distributed to each storage node, ensuring that the system resources of each storage node are relatively balanced. By updating the storage location of each data to be stored on the initial hash ring, the problems of excessive data movement, data skew caused by a single evaluation dimension, and load imbalance between storage nodes in the traditional algorithm during data sharding are solved, thereby improving data sharding efficiency and system performance.

[0258] As an implementation of the present application, in order to describe in detail the clustering of each data to be stored, the target clustering result determination module 420 may further specifically include:

[0259] a cluster node determination unit, configured to determine a plurality of cluster nodes based on each of the to-be-stored data, the quantity of each of the to-be-stored data, and a preset number of clusters;

[0260] A first Euclidean distance determining unit, configured to calculate a first Euclidean distance between each of the to-be-stored data and each clustering node;

[0261] an initial clustering result determining unit, configured to place each of the data to be stored into the cluster corresponding to the data to be stored and having the smallest first Euclidean distance, to obtain an initial clustering result of each of the data to be stored;

[0262] The target clustering result determination unit is used to repeatedly perform the following steps until the cluster to which each of the data to be stored belongs does not change, thereby obtaining the target clustering result of each of the data to be stored: for each initial clustering result, calculating the second Euclidean distance between each of the data to be stored in the initial clustering result and the center point of each initial clustering result, and placing each of the data to be stored in the cluster with the smallest second Euclidean distance corresponding to the data to be stored.

[0263] As an implementation of the present application, in order to describe in detail the repeated clustering process of the target clustering result determination unit on the initial clustering result, the target clustering result determination unit may further specifically include:

[0264] a center point determination subunit, configured to determine, for each initial clustering result, a center point of the initial clustering result based on a feature vector corresponding to each of the data to be stored in the initial clustering result and the number of each of the data to be stored in the initial clustering result;

[0265] A second Euclidean distance determination subunit, configured to respectively calculate a second Euclidean distance between each of the to-be-stored data and the center point of each initial clustering result;

[0266] The clustering subunit is configured to place each of the data to be stored into the cluster with the smallest second Euclidean distance corresponding to the data to be stored.

[0267] In one example, for each of the to-be-stored data in each target clustering result, each of the to-be-stored data has a preset number of feature vectors.

[0268] As an implementation of the present application, in order to describe in detail the determination of the mean of each eigenvector in the target clustering result, the mean determination module 430 may further specifically include:

[0269] The mean determination unit is configured to determine, for each current target clustering result, a mean of each eigenvector in the current target clustering result based on a eigenvector value corresponding to each eigenvector of the data to be stored in the current target clustering result.

[0270] As an implementation of the present application, in order to describe in detail a method for determining a resource occupancy index of each data to be stored, the resource occupancy index determining module 440 may further specifically include:

[0271] A ranking result determining unit, configured to obtain a ranking result of each eigenvector in each target clustering result by sorting the eigenvectors in each target clustering result based on the mean of each eigenvector in the target clustering result;

[0272] A resource occupancy index determination unit is used to determine, for each current target clustering result, the resource occupancy index of each data to be stored in the current target clustering result based on the ranking result of each feature vector in the current target clustering result and the preset weight of each feature vector in the current target clustering result.

[0273] As an implementation of the present application, in order to describe in detail the storage process of each data to be stored in the initial hash ring, the above-mentioned device may further include:

[0274] A primary key acquisition module is used to obtain the primary key of each data to be stored;

[0275] A hash code value determination module is used to calculate the hash code value of the primary key to obtain the hash code value corresponding to each data to be stored;

[0276] A hash value determination module, configured to perform hash calculation on the hash code value to obtain a hash value corresponding to each of the data to be stored;

[0277] A storage module is used to distribute each of the to-be-stored data to a corresponding storage node on the initial hash ring based on the hash value.

[0278] As an implementation of the present application, in order to describe in detail the updating of the storage location of each to-be-stored data on the initial hash ring, the data shard storage module 450 may further include:

[0279] A node acquisition unit, configured to acquire each storage node on the initial hash ring and a storage weight corresponding to each storage node;

[0280] A data shard storage unit is used to update the storage location of each data to be stored on the initial hash ring based on the resource occupancy index of each data to be stored and the storage weight, so that the difference in storage weight between each storage node is less than or equal to a preset difference threshold.

[0281] The data sharding device provided in the embodiment of the present application can be used to execute the data sharding method provided in the above-mentioned method embodiments. Its implementation principles and technical effects are similar, and for the sake of simplicity, they will not be repeated here.

[0282] Based on the same inventive concept, an embodiment of the present application also provides an electronic device.

[0283] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 9 As shown, the electronic device may include a processor 501 and a memory 502 storing computer programs or instructions.

[0284] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0285] The memory 502 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 502 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory. In a specific embodiment, the memory 502 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0286] The processor 501 implements any one of the data sharding methods in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0287] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 9 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.

[0288] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or devices in the embodiment of the present invention.

[0289] Bus 510 comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 510 can comprise one or more buses.Although the embodiment of the present invention describes and shows specific bus, the present invention considers any suitable bus or interconnection.

[0290] The electronic device can execute the data slicing method in the embodiment of the present invention, thereby achieving Figure 4-Figure 7 Any of the described data sharding methods.

[0291] In addition, in combination with the data sharding method in the above embodiment, the embodiment of the present invention can provide a readable storage medium for implementation. The readable storage medium stores program instructions; when the program instructions are executed by a processor, any one of the data sharding methods in the above embodiment is implemented.

[0292] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0293] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in unit, a function card or the like. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0294] It should also be noted that the exemplary embodiments described herein describe methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the steps described above. In other words, the steps may be performed in the order described in the embodiments, or in a different order, or several steps may be performed simultaneously.

[0295] The above description is only a specific embodiment of the present invention. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention.

Claims

1. A data sharding method, characterized in that: The method comprises: Obtain multiple data to be stored on the initial hash ring; Clustering the data to be stored to obtain target clustering results for the data to be stored; For each target clustering result, calculating the mean of each eigenvector in the target clustering result; For each target clustering result, based on the mean and the preset weight of each eigenvector in the target clustering result, determining the resource occupation index of each data to be stored, including: Based on the mean of each eigenvector in each target clustering result, sorting is performed according to the size of the mean to obtain a ranking result of each eigenvector in each target clustering result; For each current target clustering result, based on the ranking result of each feature vector in the current target clustering result and the preset weight of each feature vector in the current target clustering result, determine the resource occupation index of each data to be stored in the current target clustering result; Based on the resource occupation index of each of the data to be stored, updating the storage location of each of the data to be stored on the initial hash ring includes: Obtain each storage node on the initial hash ring and the storage weight corresponding to each storage node; Based on the resource occupancy index of each data to be stored and the storage weight, the storage location of each data to be stored on the initial hash ring is updated so that the difference in storage weight between each storage node is less than or equal to a preset difference threshold.

2. The method according to claim 1, characterized in that The clustering of the data to be stored to obtain target clustering results of the data to be stored includes: Determining a plurality of cluster nodes based on each of the to-be-stored data, the quantity of each of the to-be-stored data, and a preset number of clusters; Calculating the first Euclidean distance between each of the data to be stored and each cluster node; placing each of the data to be stored into the cluster corresponding to the data to be stored and having the smallest first Euclidean distance, to obtain an initial clustering result for each of the data to be stored; Repeat the following steps until the cluster to which each data to be stored belongs does not change, and obtain the target clustering result of each data to be stored: For each initial clustering result, the second Euclidean distance between each data to be stored in the initial clustering result and the center point of each initial clustering result is calculated, and each data to be stored is placed in the cluster with the smallest second Euclidean distance corresponding to the data to be stored.

3. The method according to claim 2, characterized in that The step of calculating, for each initial clustering result, a second Euclidean distance between each of the to-be-stored data in the initial clustering result and the center point of each initial clustering result, and placing each of the to-be-stored data into a cluster corresponding to the to-be-stored data and having the smallest second Euclidean distance, includes: For each initial clustering result, based on the feature vectors corresponding to the data to be stored in the initial clustering result and the number of the data to be stored in the initial clustering result, determine the center point of the initial clustering result; Calculating the second Euclidean distance between each of the to-be-stored data and the center point of each initial clustering result respectively; Each of the data to be stored is placed into the cluster corresponding to the data to be stored and having the smallest second Euclidean distance.

4. The method according to claim 1, wherein For each of the to-be-stored data in each target clustering result, each of the to-be-stored data has a preset number of feature vectors; The step of calculating the mean of each eigenvector in each target clustering result includes: For each current target clustering result, based on the eigenvector values ​​corresponding to the eigenvectors of the data to be stored in the current target clustering result, a mean value of each eigenvector in the current target clustering result is determined.

5. The method according to claim 1, wherein Before obtaining the plurality of data to be stored on the initial hash ring, the method further includes: Get the primary key of each data to be stored; Calculating a hash code value for the primary key to obtain a hash code value corresponding to each of the data to be stored; Performing hash calculation on the hash code value to obtain a hash value corresponding to each of the data to be stored; Based on the hash value, each of the data to be stored is allocated to a corresponding storage node on the initial hash ring.

6. A data sharding device, characterized in that: The device comprises: An information acquisition module is used to acquire a plurality of data to be stored on the initial hash ring, and each feature vector corresponding to each of the data to be stored; a target clustering result determination module, configured to cluster the data to be stored to obtain a target clustering result for the data to be stored; A mean determination module, configured to calculate, for each target clustering result, the mean of each eigenvector in the target clustering result; a resource occupancy index determination module, configured to determine, for each target clustering result, a resource occupancy index of each of the data to be stored based on the mean and a preset weight of each eigenvector in the target clustering result; The resource occupancy index determination module is specifically used to: Based on the mean of each eigenvector in each target clustering result, sorting is performed according to the size of the mean to obtain a ranking result of each eigenvector in each target clustering result; For each current target clustering result, based on the ranking result of each feature vector in the current target clustering result and the preset weight of each feature vector in the current target clustering result, determine the resource occupation index of each data to be stored in the current target clustering result; A data shard storage module, configured to update a storage location of each of the data to be stored on the initial hash ring based on a resource occupancy indicator of each of the data to be stored; The data sharding storage module is specifically used to: Obtain each storage node on the initial hash ring and the storage weight corresponding to each storage node; Based on the resource occupancy index of each data to be stored and the storage weight, the storage location of each data to be stored on the initial hash ring is updated so that the difference in storage weight between each storage node is less than or equal to a preset difference threshold.

7. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data sharding method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the data sharding method as described in any one of claims 1 to 5 are implemented.

Images