Data library separation method, device, equipment, storage medium and program product

Abstract

The application discloses a data library distribution method and device, equipment, a storage medium and a program product. The method comprises the following steps: extracting a target field value from a target message to be processed based on a preconfigured target field, and converting the target field value into a first hash code with a specified length; determining a target hash algorithm corresponding to the target field; combining the high-order element and the low-order element of the first hash code by using the target hash algorithm to generate a second hash code; and determining a target database to be distributed based on the second hash code. The second hash code integrates high-order features and low-order features, which can fully reflect the randomness of the overall hash result, improve the hash dispersion, balance the overall access pressure of the database cluster, reduce the situation that a single machine is shut down due to overload, and reduce resource waste caused by the fact that some machines receive less access.

Description

Technical Field

[0001] This application relates to the field of big data intelligent analysis technology, and in particular to a data partitioning method, a data partitioning device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In the order systems of online shopping malls, banks, and other payment institutions, database clusters are typically deployed to handle massive, high-concurrency, low-latency payment transaction requests. This reduces the size of data stored in a single database and improves the speed of order data retrieval.

[0003] To ensure consistency in database access for the same transaction request across multiple databases in a multi-database scenario, horizontal database sharding is achieved by selecting a unique identifier from the transaction and calculating the database number to be accessed using modulo. This approach guarantees the uniqueness of hash results generated for the same order number and the randomness of hash results generated for different order numbers. However, during the modulo operation, because the number of database channels in the database cluster is far less than the value range of a 32-bit hash mapping, the higher-order bits of the calculated result are discarded, and the hash result depends only on the lower-order bits. Consequently, the calculated database number to be accessed does not fully reflect the randomness of the hash algorithm result, resulting in poor dispersion performance. Summary of the Invention

[0004] This application provides a data sharding method, apparatus, device, storage medium, and program product to solve the problem that the hash results calculated by the hash algorithm mentioned in the related technology only depend on the low-order value, which cannot fully reflect the randomness of the hash algorithm results and has poor dispersion index performance.

[0005] According to a first aspect of this application, a data sharding method is provided, the method comprising:

[0006] Extract target field values ​​from the target message to be processed based on pre-configured target fields, and convert the target field values ​​into a first hash code of a specified length;

[0007] Determine the target hash algorithm corresponding to the target field;

[0008] The target hash algorithm is used to combine the high-order and low-order elements of the first hash code to generate the second hash code;

[0009] The target database to be distributed is determined based on the second hash code.

[0010] According to a second aspect of this application, a data partitioning apparatus is provided, the apparatus comprising:

[0011] The first hash code acquisition module is used to extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length;

[0012] A target hash algorithm determination module is used to determine the target hash algorithm corresponding to the target field;

[0013] The second hash code acquisition module is used to combine the high-order and low-order elements of the first hash code using the target hash algorithm to generate the second hash code.

[0014] The target database determination module is used to determine the target database to be distributed based on the second hash code.

[0015] According to a third aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect described above.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method of the first aspect described above.

[0020] According to a fifth aspect of this application, a computer program product is provided, the computer program product including computer-executable instructions, which, when executed, are used to implement the method of the first aspect described above.

[0021] In this embodiment, after obtaining the first hash code of the target field value, the high-order and low-order elements of the first hash code are combined using a matching target hash algorithm to generate a second hash code. Since the second hash code incorporates both the high-order and low-order elements of the first hash code, the modulo operation obtained from the second hash code also incorporates both the high-order and low-order elements of the hash code. This allows the target database determined based on the modulo result to fully reflect the overall randomness of the hash result while also improving the hash dispersion, thereby achieving load balancing of the overall access pressure of the database cluster. This reduces the possibility of a single machine crashing due to overload and also reduces resource waste caused by some machines receiving fewer accesses. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an embodiment of a data partitioning method provided in Embodiment 1 of this application;

[0024] Figure 2 This is a flowchart of an embodiment of a data partitioning method provided in Embodiment 2 of this application;

[0025] Figure 3 This is a flowchart of an embodiment of a data partitioning method provided in Embodiment 3 of this application;

[0026] Figure 4 This is a flowchart of an embodiment of a data partitioning method provided in Embodiment 4 of this application;

[0027] Figure 5 This is a structural block diagram of an embodiment of a data sharding device provided in Embodiment 5 of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] In addition, it should be noted that the acquisition, storage, use, and processing of data in the following embodiments of this application all comply with the relevant provisions of national laws and regulations.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating an embodiment of a data sharding method provided in Embodiment 1 of this application. This embodiment can be applied to a server. Figure 1 As shown, this embodiment may include the following steps:

[0034] Step 101: Extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length.

[0035] In this context, the target message refers to the message to be processed. The target message varies depending on the application scenario, and this embodiment does not impose any limitations on it. For example, in an e-commerce scenario, the target message can be an order message; in a banking system, the target message can be a user-triggered action message or other request message.

[0036] The target field can be a field selected from the fields of data tables stored in various databases. Once the target field is determined, the corresponding target field value can be extracted from the target message to be processed based on the target field. For example, the target field may include order number, user identification number, etc.

[0037] After extracting the target field value from the target message, the target field value can be converted into a first hash code of a specified length. This first hash code can be represented as a binary number of type int32, with a specified length of 32 bits.

[0038] In one implementation, a pre-defined hash algorithm can be used to convert the target field value into a binary hash code, which serves as the first hash code. For example, assuming the target field value is an order number, which can be composed of "fixed value + current timestamp + random string", the first hash code corresponding to the order number can be calculated as follows:

[0039] s[0]*31^(n-1)+s[1]*31^(n-2)+...+s[n-1]

[0040] Where s[i] is the i-th character of the order number, and n is the length of the order number.

[0041] This example uses the algorithm described above to calculate the first hash code, which ensures the randomness of generating hash codes for different order numbers (different target field values).

[0042] Step 102: Determine the target hash algorithm corresponding to the target field.

[0043] Hash algorithms are used to transform inputs of arbitrary length into outputs of fixed length through hashing. The output is the hash result. Simply put, a hash algorithm is a function that compresses a message of arbitrary length into a message digest of a certain fixed length.

[0044] During implementation, multiple hash algorithms can be pre-generated for selection. Once the target field value is determined, the most suitable hash algorithm can be selected from multiple hash algorithms based on the target field value as the target hash algorithm.

[0045] For example, a corresponding field list can be generated for each hash algorithm, and then the current target field can be found in the field list of each hash algorithm, so that the hash algorithm corresponding to the field list where the target field is located is used as the target hash algorithm.

[0046] Step 103: The high-order and low-order elements of the first hash code are combined using the target hash algorithm to generate the second hash code.

[0047] In practice, when performing database sharding, after obtaining the hash code, a modulo operation can be performed based on the number of databases in the database cluster to determine the hash result (if the result is negative, its absolute value needs to be taken). Then, the database number to be accessed is determined based on the hash result, achieving horizontal database sharding. However, if database sharding is performed based on the first hash code generated in step 101, since the number of databases in the database cluster is far less than the range of the 32-bit hash mapping, the high-order data characteristics of the calculation result will be discarded, and the calculated hash result will only depend on the low-order values. Therefore, the calculated database number to be accessed cannot fully reflect the overall randomness of the hash result, resulting in poor dispersion performance. For example, Table 1 shows the relationship between the order number, the first hash code, and the hash result after performing a modulo operation on the first hash code (i.e., the modulo 16 result in Table 1):

[0048]

[0049]

[0050] Table 1

[0051] As shown in Table 1, there is a unique correspondence between different order numbers and their calculated first hash codes, and the distribution of the hash codes exhibits randomness. However, when performing a modulo-16 operation (assuming there are 16 databases in the database cluster), the result (i.e., the hash result) depends only on the lower 0 to 3 bits of the first hash code, a mere 4-bit value. Therefore, when attempting to calculate the database number using the modulo operation result, because bits 4 to 32 of the first hash code are not involved in the modulo operation, the uniform distribution of the hash result cannot be guaranteed, as shown in Table 2.

[0052] First hash The result modulo 16 1000011100011110110110000100001 0001 0000110100011010111100100100001 0001 1111100010010011010101101100001 0001

[0053] Table 2

[0054] In Table 2, the three completely different first hash codes yield the same hash result after modulo operation. This means that in the current production situation, these three completely different first hash codes calculated from different order numbers will ultimately access database 1 (0001 represents database 1), and the hash dispersion after modulo operation cannot be guaranteed.

[0055] Based on this, in step 103, the target hash algorithm is used to combine the high-order and low-order elements of the first hash code to generate the second hash code. This results in a hash result obtained by taking the modulus of the second hash code, which incorporates the high-order and low-order elements of the first hash code, thereby improving the hash dispersion.

[0056] In one implementation, the target hash algorithm can use one or a combination of methods such as scaling, truncation, XOR calculation, and multiplication to combine the high-order and low-order elements of the first hash code to generate the second hash code.

[0057] Step 104: Determine the target database to be distributed based on the second hash code.

[0058] In one implementation, after obtaining the second hash code, a modulo operation can be performed based on the second hash code and the number of databases in the database cluster, and the target database to be distributed can be determined according to the hash result obtained after the modulo operation. Of course, other methods can also be used to determine the target database based on the second hash code, and this embodiment does not limit this.

[0059] In this embodiment, after obtaining the first hash code of the target field value, the high-order and low-order elements of the first hash code are combined using a matching target hash algorithm to generate a second hash code. Since the second hash code incorporates both the high-order and low-order elements of the first hash code, the modulo operation obtained from the second hash code also incorporates both the high-order and low-order elements of the hash code. This allows the target database determined based on the modulo result to fully reflect the overall randomness of the hash result while also improving the hash dispersion, thereby achieving load balancing of the overall access pressure of the database cluster. This reduces the possibility of a single machine crashing due to overload and also reduces resource waste caused by some machines receiving fewer accesses.

[0060] Example 2

[0061] Figure 2 This is a flowchart illustrating a data partitioning method embodiment provided in Embodiment 2 of this application. Based on Embodiment 1, this embodiment describes the process of determining the target hash algorithm and the process of evaluating hash dispersion, as follows... Figure 2 As shown, this embodiment may include the following steps:

[0062] Step 201: Collect the corresponding field value from the sample message for the given field.

[0063] The sample messages may include pre-collected historical messages, and the given field can be a field pre-configured by the user. For a given field, the field value corresponding to that given field can be extracted from each sample message.

[0064] Step 202: Convert each of the field values ​​into a first sample hash code of a specified length.

[0065] In one implementation, the extracted field value can be converted into a first sample hash code of a specified length using the method described in step 101 of Embodiment 1. The first sample hash code can be exemplarily represented as a binary number of type int32, with a specified length of 32 bits.

[0066] Step 203: Using different existing hash algorithms, the high-order and low-order elements of the first sample hash code are combined to generate the second sample hash code.

[0067] In implementation, multiple different hash algorithms can be pre-generated. After extracting the field values ​​of a given field from different sample messages and calculating the first sample hash code corresponding to each field value, for each first sample hash code, different existing hash algorithms are used to transform it to obtain the second sample hash code corresponding to each hash algorithm. Thus, a first sample hash code has multiple different second sample hash codes.

[0068] Step 204: Determine the database of the corresponding sample message based on the second sample hash code.

[0069] In this step, a modulo operation is performed on the hash codes of each second sample, and the corresponding database is determined based on the modulo result. Therefore, for a single sample message, the calculated corresponding database may differ depending on the hash algorithm used.

[0070] Step 205: Based on the database of sample messages assigned to each hash algorithm for the given field, a pre-generated standard deviation-based hash evaluation algorithm is used to evaluate the dispersion of each hash algorithm and obtain the evaluation value corresponding to each hash algorithm.

[0071] One important metric for evaluating the quality of hash algorithms is dispersion, which is the proportion of the hash result distribution to the range of hash output values. The higher this proportion, the better the dispersion and the better the hash algorithm design. In database sharding scenarios, better hash algorithms have better dispersion, which can make the data more evenly distributed across different databases, thereby improving the efficiency of database usage.

[0072] In this step, a pre-created hash evaluation algorithm can be used to evaluate the dispersion of each hash algorithm, and the evaluation value of the dispersion of each hash algorithm is obtained. This evaluation value is used to represent the quality of the hash algorithm and is a description of the hash distribution.

[0073] Wherein, the hash evaluation algorithm is a standard deviation-based hash evaluation algorithm, then in one embodiment, step 205 may further include the following steps:

[0074] For each hash algorithm, an array of a set length is declared to store the number of hits in each database calculated by the hash algorithm for each sample message; the standard deviation is calculated based on the value of each element in the array, which serves as an evaluation value of the dispersion of the hash algorithm.

[0075] In one implementation, the length of the array (i.e., the set length) is equal to the number of databases in the database cluster. For example, if the number of databases in the database cluster is 16, then the length of the array is also 16. If the database numbers are 0-15, then the indices of the array elements in the array are also 0-15.

[0076] For example, assuming the given field is order number, and there are 5 sample data, then there are 5 different order numbers. That is, each hash algorithm is executed 5 times. The second sample hash code, hash result (modulo 16 in the case of 16 databases), and corresponding array index result for each order number calculated by a certain hash algorithm are shown in Table 3:

[0077] Execution count Sample message order number Second sample hash Modulo 16 results Array subscript 1 0130166434774793938309 1101111100001111011001000111101 1101 Res

[13] 2 0130166434774793915422 1101111011010100010101000010110 0110 Res[6] 3 0130165641115923555301 1101111110110101101111100111011 1011 Res

[11] 4 0130165641115923585276 1101111101111010101001101011011 1011 Res

[11] 5 0130166434774793924363 1101111011101111100100111110100 0100 Res[4]

[0078] Table 3

[0079] Based on Table 3, among the 5 sample messages, database number 13 (i.e., array index 13, the same below) was hit once, database number 6 was hit once, database number 11 was hit twice, and database number 4 was hit once. Therefore, for the current hash algorithm, the contents recorded in the array of length 16 are as follows:

[0080] [0,0,0,0,1,0,1,0,0,0,0,2,0,1,0,0]

[0081] To facilitate subsequent calculations of variance, standard deviation, etc., and to improve calculation accuracy, array elements can be represented using double-precision floating-point type (double), that is:

[0082] [0.0,0.0,0.0,0.0,1.0,0.0,1.0,0.0,0.0,0.0,0.0,2.0,0.0,1.0,0.0,0.0]

[0083] After obtaining the above array, the standard deviation can be calculated based on the values ​​of each element in the array. In implementation, the standard deviation is calculated as follows:

[0084] Calculate the sum of all elements in the array; calculate the mean of all elements in the array based on the sum; calculate the variance of all elements in the array based on the mean; calculate the standard deviation of all elements in the array based on the variance.

[0085] For example, in the array example above, the sum of all element values ​​in the array is 5 (1+1+2+1=5); the average of all element values ​​in the array is the ratio of the sum to the number of elements, i.e., 5 / 16=0.3125; the variance is 0.3398437500000001; then the standard deviation is 0.582961190818051; that is, the dispersion assessment value of the current hash algorithm is 0.582961190818051.

[0086] Similarly, the dispersion evaluation values ​​of other hashing algorithms can be calculated using the method described above.

[0087] Step 206: Add the given field to the field list corresponding to the hash algorithm with the largest evaluation value, and finally obtain the field list corresponding to each hash algorithm.

[0088] Once the evaluation values ​​of each hash algorithm are obtained, the hash algorithm with the highest evaluation value can be identified, and then the currently given field can be added to the field list of the hash algorithm with the highest evaluation value.

[0089] By repeating the above process, when there are enough given fields, a list of fields for each hash algorithm can be obtained.

[0090] Step 207: Extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length.

[0091] Step 208: Match the target field in the field list of each hash algorithm, and use the hash algorithm corresponding to the field list in which the target field is located as the target hash algorithm.

[0092] Step 209: Use the target hash algorithm to combine the high-order and low-order elements of the first hash code to generate the second hash code.

[0093] Step 210: Determine the target database to be distributed based on the second hash code.

[0094] In this embodiment, the dispersion of each hash algorithm is evaluated based on a standard deviation-based hash evaluation algorithm. For a given field, the dispersion evaluation value of each hash algorithm can be obtained based on the database sharding results of sample messages for that given field. Then, based on this evaluation value, the optimal hash algorithm suitable for the given field is determined, and the given field is further added to the field list corresponding to the optimal hash algorithm. Subsequently, for a target field, by matching it with the field lists of each hash algorithm, the optimal hash algorithm for that target field can be found, thereby improving the hashability of database sharding and achieving load balancing of database cluster access in high-concurrency scenarios.

[0095] Example 3

[0096] Figure 3 This is a flowchart illustrating a data partitioning method embodiment provided in Embodiment 3 of this application. Based on Embodiment 1 or Embodiment 2, this embodiment describes the process of obtaining the second hash code. In this embodiment, the target hash algorithm used is a hash algorithm that first overflows and then truncates.

[0097] like Figure 3 As shown, this embodiment may include the following steps:

[0098] Step 301: Extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length.

[0099] The specified length is 32 bits.

[0100] Step 302: Determine the target hash algorithm corresponding to the target field.

[0101] Step 303: Determine the adjusted hash code corresponding to the first hash code.

[0102] The function of adjusting the hash code is to merge the high and low bits of the first hash code.

[0103] In one implementation, the first hash code can be used as the adjustment hash code. In other implementations, a preset special hash code can also be used as the adjustment hash code, for example, the special hash code is 0x61c88888.

[0104] The introduction of the special hash code 0x61c88888 embodies the idea of ​​the Fibonacci sequence (also known as the golden ratio sequence). 0x61c88888 is a hexadecimal value that approximates the maximum value that a 32-bit unsigned binary number can represent (2^32-1), multiplied by 0.618 (the golden ratio point), and then the absolute value of the result after representing it as a signed integer.

[0105] Step 304: Calculate the product of the first hash code and the adjusted hash code, and overflow the product to obtain a 32-bit hash code, which is used as the second hash code.

[0106] In one embodiment, if the hash code is adjusted to the first hash code, and the length of the first hash code is 32 bits, then multiplying the first hash code by the first hash code yields a 64-bit hash code. However, since the maximum binary number that a 32-bit integer can represent is 32 bits, the product of the two hash codes is 64 bits, which exceeds the maximum of 32 bits. Therefore, the excess part will automatically overflow and be discarded.

[0107] For example, the hash codes of three random orders are multiplied to obtain a 64-bit hash code, and the results of automatic overflow are shown in Table 4:

[0108]

[0109] Table 4

[0110] In Table 4, when the hash code is represented using int32, the high 32 bits of the complete 64-bit result (i.e., the bolded part in Table 4) will automatically overflow and be discarded, and the second hash code obtained after truncation is the low 32 bits of the 64-bit result.

[0111] In another embodiment, if the hash code is adjusted to the special hash code, and the length of the first hash code is 32 bits, then multiplying the first hash code by the special hash code yields a 64-bit hash code. However, since the maximum binary number that a 32-bit integer can represent is 32 bits, the product of the two hash codes is 64 bits, which exceeds the maximum of 32 bits. Therefore, the excess part will automatically overflow and be discarded.

[0112] For example, the hash codes of three random orders are multiplied by a special hash code to obtain a 64-bit hash code, and the results of automatic overflow are shown in Table 5:

[0113]

[0114] Table 5

[0115] In Table 5, when the hash code is represented using int32, the high 32 bits of the complete 64-bit result (i.e., the bolded part in Table 5) will automatically overflow and be discarded, and the second hash code obtained after truncation is the low 32 bits of the 64-bit result.

[0116] Step 305: Determine the corresponding binary representation of each database number in the database cluster, and use the number of bits in the binary representation as the extraction bit.

[0117] For example, if there are 16 databases in the database cluster, numbered 0-15, then the corresponding binary representation is 4 bits, which means the number of bits extracted is 4 bits.

[0118] Step 306: Calculate the difference between the extracted bit length and the length of the second hash code.

[0119] For example, assuming the number of bits extracted is 4 and the length of the second hash code is 32, the difference is 28.

[0120] Step 307: Shift the second hash code to the right by the number of bits corresponding to the difference, and extract the value corresponding to the lower bits of the extracted bits, and determine the target database based on this value.

[0121] Because the characteristics of the product operation cause the features of the hash code to be concentrated in the middle part of the result, the overall features of the hash code are mainly mapped to the position of the header of the overflowed int32-bit result value. Therefore, by right-shifting the second hash code and truncating the low-order values, the hash result is obtained, and then the target database is determined based on the hash result.

[0122] For example, assuming the difference is 28, the second hash code is shifted right by 28 bits, and then the values ​​of bits 0-3 are used to determine the target database. For instance, after shifting the second hash code in Table 4 right by 28 bits, the hash codes shown in Table 6 are obtained:

[0123]

[0124] Table 6

[0125] In Table 6, the target database corresponding to the first order number is

[0101] , which is the database numbered 5; the target database corresponding to the second order number is

[1101] , which is the database numbered 13; and the target database corresponding to the third order number is

[0011] , which is the database numbered 3.

[0126] It should be noted that, in the two methods of determining the adjustment hash code mentioned above, using a special hash code as the adjustment hash code results in fewer features being lost during the overflow process compared to using the first hash code as the adjustment hash code. This allows the main features of the hash code to be preserved within the int32-bit range to a greater extent, reducing automatic overflow of the adjustment bits, thereby better reducing feature loss and improving the performance of the algorithm.

[0127] This embodiment employs a hash algorithm based on multiplication. After multiplying the initially obtained hash codes and allowing overflow, the hash result is truncated to determine the target database. The product property causes the numerical result of the two numbers to be concentrated in the middle part of the result. Therefore, by first triggering automatic overflow to discard the high-order bits, and then truncating the low-order bits by right shifting, the final result obtained is the middle part of the value with more concentrated features. This can retain the main features of the hash code within the int32-bit range to a greater extent, reduce automatic overflow with correction bits, and thus better reduce feature loss and improve the performance of the algorithm.

[0128] Example 4

[0129] Figure 4 This is a flowchart illustrating a data partitioning method embodiment provided in Embodiment 4 of this application. Based on Embodiment 1 or Embodiment 2, this embodiment describes the process of obtaining the second hash code. In this embodiment, a hash algorithm based on XOR processing is employed.

[0130] like Figure 4As shown, this embodiment may include the following steps:

[0131] Step 401: Extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length.

[0132] The specified length is 32 bits.

[0133] Step 402: Determine the target hash algorithm corresponding to the target field.

[0134] Step 403: Reduce the first hash code by a set ratio so that the first hash code is shifted to the right by a set number of bits associated with the set ratio, thereby obtaining a reduced hash code.

[0135] The ratio can be a power of 2, such as 2^n. 4 2 8 2 16 2 28 In practice, during hash code reduction, all the numerical bits of the hash code are shifted to lower bits. For example, reducing the first hash code by 2... 16 Then the values ​​of bits 16 to 32 of the first hash code will be moved to bits 0 to 15, so that the low-order bits of the original bits 0 to 15 will be automatically truncated and discarded, resulting in a reduced hash code.

[0136] The following table 7 provides an example illustration of setting the ratio and the number of right shifts:

[0137]

[0138]

[0139] Table 7

[0140] In Table 7, for ease of comparison, the first empty position is filled with 0 to maintain the 32-bit length of the hash code. It can be seen that the value that was originally in the high bit of the first hash code has been shifted to a lower position to the right after being reduced by a certain factor.

[0141] Step 404: Perform an XOR operation between the reduced hash code and the first hash code to obtain the second hash code.

[0142] After obtaining the reduced hash code, it can be XORed with the first hash code, and the result becomes the second hash code. For example, if the reduced hash code is obtained by reducing the first hash code by a power of 2 (2^16), then XORing it with the first hash code is equivalent to XORing bits 16-32 of the first hash code with bits 0-15, and the result automatically overwrites the original bits 0-15. This results in a second hash code that remaps the entire first hash code to its lower bits. Subsequent modulo operations using the second hash code will ensure that all the values ​​from the first hash code participate in subsequent calculations.

[0143] For example, after performing an XOR operation on the reduced hash code obtained in Table 7 and its corresponding first hash code, the resulting second hash code and the modulo operation result can be shown in Table 8:

[0144]

[0145] Table 8

[0146] As can be seen from Table 8, the hash result (i.e., the modulo 16 result) changes after performing an XOR operation on the first hash code, and the hash result also changes depending on the degree of shrinkage in the previous step. After adopting this shrinkage and XOR approach, the entire value of the hash code ultimately participates in the final modulo operation.

[0147] Step 405: Determine the corresponding binary representation of each database number in the database cluster, and use the number of bits in the binary representation as the extraction bit.

[0148] Step 406: Extract the value corresponding to the lower bits of the second hash code, and determine the target database based on this value.

[0149] For example, in Table 8, if the number of bits extracted is 4, then after obtaining the second hash code, the values ​​of bits 0-3 are extracted as the hash result, and then the hash result is converted into a decimal number to obtain the number of the target database.

[0150] In this embodiment, a hash algorithm based on shrinking hash code and XOR operation is adopted. The initially obtained hash code is shrunk by a set ratio to obtain a shrunk hash code. Then, the shrunk hash code is XORed with the initially obtained first hash code to obtain a second hash code. The target database is determined based on the hash result of the second hash code. This makes the values ​​that were originally in the high bits of the first hash code shift to the lower right position after being shrunk by a specific factor, so that more values ​​in the hash code participate in the modulo operation. Compared with the hash result that only depends on part of the hash code (such as the low bits), the hash dispersion is greatly improved.

[0151] In an exemplary experiment, by simulating 100 million randomly generated order data points per day, statistics were compiled for four time periods: one week, one month, six months, and one year. The dispersion evaluation values ​​obtained using existing hash algorithms, the hash algorithm mentioned in Example 3, and the hash algorithm mentioned in Example 4, respectively, are shown in Table 9.

[0152]

[0153]

[0154] Table 9

[0155] As can be seen from Table 9, the hashing algorithms mentioned in Examples 3 and 4 have significantly reduced the dispersion of their hashing results compared to existing hashing algorithms. Since the dispersion of results in the dispersion evaluation algorithm is essentially variance, the smaller the value, the better the hashing dispersion and the more uniform the hashing results.

[0156] Example 5

[0157] Figure 5 A structural block diagram of a data sharding device embodiment provided in Embodiment 5 of this application may include the following modules:

[0158] The first hash code acquisition module 501 is used to extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length;

[0159] The target hash algorithm determination module 502 is used to determine the target hash algorithm corresponding to the target field;

[0160] The second hash code acquisition module 503 is used to combine the high-order and low-order elements of the first hash code using the target hash algorithm to generate the second hash code.

[0161] The target database determination module 504 is used to determine the target database to be distributed based on the second hash code.

[0162] In one embodiment, the target hash algorithm determination module 502 is specifically used for:

[0163] Get the list of fields corresponding to different hash algorithms;

[0164] The target field is matched in each of the preset field lists, and the hash algorithm corresponding to the field list in which the target field is located is used as the target hash algorithm.

[0165] In one embodiment, the apparatus may further include a field list generation module, comprising:

[0166] The field value acquisition module is used to collect the corresponding field value from the sample message for a given field;

[0167] The first sample hash code generation module is used to convert each of the field values ​​into a first sample hash code of a specified length.

[0168] The second sample hash code generation module is used to combine the high-order and low-order elements of the first sample hash code using different existing hash algorithms to generate the second sample hash code.

[0169] The database sharding module is used to determine the database corresponding to the sample message based on the second sample hash code;

[0170] The evaluation value determination module is used to evaluate the dispersion of each hash algorithm based on the database of sample messages assigned to the given field using each hash algorithm, and obtain the evaluation value corresponding to each hash algorithm by using a pre-generated standard deviation-based hash evaluation algorithm.

[0171] The field addition module is used to add the given field to the field list corresponding to the hash algorithm with the highest evaluation value.

[0172] In one embodiment, the evaluation value determination module is specifically used for:

[0173] For each hash algorithm, declare an array of a set length to store the number of hits in each database calculated by the hash algorithm for each sample message;

[0174] The standard deviation is calculated based on the values ​​of each element in the array, and used as an evaluation value for the dispersion of the hash algorithm.

[0175] In one embodiment, the specified length is 32 bits; the second hash code acquisition module 503 is specifically used for:

[0176] Determine the adjusted hash code corresponding to the first hash code;

[0177] Calculate the product of the first hash code and the adjusted hash code, and overflow the product to obtain a 32-bit hash code, which is used as the second hash code.

[0178] In one embodiment, the second hash code acquisition module 503 is further configured to:

[0179] The first hash code is used as the adjusted hash code;

[0180] or,

[0181] The preset special hash code is used as the adjustment hash code.

[0182] In one embodiment, the special hash code is 0x61c88888.

[0183] In one embodiment, the target database determination module 504 is specifically used for:

[0184] The binary representation of each database in the database cluster is determined based on its database number, and the number of bits in the binary representation is used as the extraction bits.

[0185] Calculate the difference between the extracted bit length and the length of the second hash code;

[0186] The second hash code is shifted right by the number of bits corresponding to the difference, and the value corresponding to the lower bits of the extracted bits is extracted. The target database is determined based on this value.

[0187] In another embodiment, the second hash code acquisition module 503 is specifically used for:

[0188] The first hash code is reduced by a predetermined ratio, so that the first hash code is shifted to the right by a predetermined number of bits associated with the predetermined ratio, to obtain a reduced hash code.

[0189] The reduced hash code is XORed with the first hash code to obtain the second hash code.

[0190] In another embodiment, the target database determination module 504 is specifically used for:

[0191] The binary representation of each database in the database cluster is determined based on its database number, and the number of bits in the binary representation is used as the extraction bits.

[0192] The value corresponding to the lower bits of the second hash code is extracted, and the target database is determined based on this value.

[0193] The data partitioning device provided in this application embodiment can execute the methods in the above method embodiments, and has the corresponding functional modules and beneficial effects of executing the methods.

[0194] Example 6

[0195] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the methods of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0196] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0197] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0198] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the steps described in the above method embodiments.

[0199] In some embodiments, the steps described in the above method embodiments may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the steps described in the above method embodiments may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the steps described in the above method embodiments by any other suitable means (e.g., by means of firmware).

[0200] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0201] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0202] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0203] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0204] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0205] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0206] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data library partitioning method, characterized by, The method includes: Extract target field values ​​from the target message to be processed based on pre-configured target fields, and convert the target field values ​​into a first hash code of a specified length; Determine the target hash algorithm corresponding to the target field; The target hash algorithm is used to combine the high-order and low-order elements of the first hash code to generate the second hash code; The target database to be distributed is determined based on the second hash code; The method for determining the target hash algorithm corresponding to the target field includes: Get the list of fields corresponding to different hash algorithms; The target field is matched in each of the preset field lists, and the hash algorithm corresponding to the field list in which the target field is located is used as the target hash algorithm; Wherein, the specified length is 32 bits; the step of combining the high-order and low-order elements of the first hash code using the target hash algorithm to generate the second hash code includes: Determine the adjusted hash code corresponding to the first hash code; Calculate the product of the first hash code and the adjusted hash code, and overflow the product to obtain a 32-bit hash code, which is used as the second hash code.

2. The method according to claim 1, characterized in that, The field lists for each hash algorithm are generated as follows: Collect the corresponding field value from the sample message for a given field; Each of the aforementioned field values ​​is converted into a first sample hash code of a specified length; Different existing hashing algorithms are used to combine the high-order and low-order elements of the first sample hash code to generate the second sample hash code. The database corresponding to the sample message is determined based on the second sample hash code; Based on the database of sample messages assigned to each hash algorithm for a given field, a pre-generated hash evaluation algorithm based on standard deviation is used to evaluate the dispersion of each hash algorithm and obtain the evaluation value corresponding to each hash algorithm. Add the given field to the list of fields corresponding to the hash algorithm with the highest evaluation value.

3. The method according to claim 2, characterized in that, The database, which assigns sample messages to a given field based on various hash algorithms, uses a pre-generated standard deviation-based hash evaluation algorithm to assess the dispersion of each hash algorithm, obtaining an evaluation value for each hash algorithm, including: For each hash algorithm, declare an array of a set length to store the number of hits in each database calculated by the hash algorithm for each sample message; The standard deviation is calculated based on the values ​​of each element in the array, and used as an evaluation value for the dispersion of the hash algorithm.

4. The method according to claim 1, characterized in that, Determining the adjusted hash code corresponding to the first hash code includes: The first hash code is used as the adjusted hash code; or, The preset special hash code is used as the adjustment hash code.

5. The method according to claim 1, characterized in that, The step of determining the target database based on the second hash code includes: The binary representation of each database in the database cluster is determined based on its database number, and the number of bits in the binary representation is used as the extraction bits. Calculate the difference between the extracted bit length and the length of the second hash code; The second hash code is shifted right by the number of bits corresponding to the difference, and the value corresponding to the lower bits of the extracted bits is extracted. The target database is determined based on this value.

6. The method according to any one of claims 1-3, characterized in that, The step of combining the high-order and low-order elements of the first hash code using the target hash algorithm to generate the second hash code further includes: The first hash code is reduced by a predetermined ratio, so that the first hash code is shifted to the right by a predetermined number of bits associated with the predetermined ratio, to obtain a reduced hash code. The reduced hash code is XORed with the first hash code to obtain the second hash code.

7. The method according to claim 6, characterized in that, The step of determining the target database to be distributed based on the second hash code includes: The binary representation of each database in the database cluster is determined based on its database number, and the number of bits in the binary representation is used as the extraction bits. The value corresponding to the lower bits of the second hash code is extracted, and the target database is determined based on this value.

8. A data sharding device, characterized in that, The device includes: The first hash code acquisition module is used to extract the target field value from the target message to be processed based on the pre-configured target field, and convert the target field value into a first hash code of a specified length; A target hash algorithm determination module is used to determine the target hash algorithm corresponding to the target field; The second hash code acquisition module is used to combine the high-order and low-order elements of the first hash code using the target hash algorithm to generate the second hash code. The target database determination module is used to determine the target database to be distributed based on the second hash code; Specifically, the target hash algorithm determination module is used to: obtain a list of fields corresponding to different hash algorithms; match the target field in each preset field list respectively, and use the hash algorithm corresponding to the field list in which the target field is located as the target hash algorithm; Wherein, the specified length is 32 bits; the second hash code acquisition module is specifically used to: determine the adjustment hash code corresponding to the first hash code; calculate the product of the first hash code and the adjustment hash code, and overflow the product to obtain a 32-bit hash code as the second hash code.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-7.

11. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1-7.

Images