A database cluster computing task allocation method and device, equipment and medium

Abstract

The application discloses a database cluster computing task allocation method and device, equipment and medium. Including: configuring the total number of hash fragments, and determining the basic binding rule between the preset hash fragment and the computing node; obtaining the number of computing nodes corresponding to the current database cluster computing resource, and determining the judgment result of secondary splitting according to the number of computing nodes and the total number of hash fragments; when secondary splitting is needed, the number of equally divided fragments is determined according to the number of computing nodes and the total number of hash fragments, and a fragment list is generated based on the number of equally divided fragments; and the computing task allocation is performed according to the fragment list. By configuring the total number of hash fragments and determining the basic binding rule, the computing cluster specification adaptation range can be expanded. Whether secondary splitting is needed can be accurately defined, and redundant operations or computing inclination can be avoided. When secondary splitting is needed, the number of equally divided fragments is determined and the fragment list is generated, the corresponding relationship between the equally divided fragments and the secondary splitting fragments and the nodes is clear, and the computing performance and resource utilization are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of databases, and in particular to a database cluster computing task allocation method, device, equipment and medium. BACKGROUND

[0002] In a distributed database system, hash sharding technology is a key means to achieve efficient data storage and balanced allocation of computing tasks. By dispersing data according to a hash algorithm to different shards, and then processing by corresponding computing nodes, the parallel processing capability and resource utilization of the system can be significantly improved.

[0003] Currently, the binding of hash shards and computing nodes in a distributed database mostly uses a static configuration method, that is, a fixed total number of hash shards is pre-set, and each shard is bound to a specific computing node, and the binding relationship remains unchanged during system operation. The allocation of computing tasks is directly based on the initial binding rules, and the shard data allocated to the corresponding node is processed.

[0004] The static binding method of the prior art is difficult to adapt to the dynamic changes in the number of computing nodes. When the number of nodes increases or decreases, if it is not reconfigured, it is easy to cause imbalance in the allocation of shards and nodes, some nodes are overloaded and some nodes are idle, which reduces the overall efficiency of the system. And lack of secondary splitting mechanism, when the total number of hash shards does not match the number of nodes, it is impossible to achieve load balancing by dynamically adjusting the shard list, it needs to rely on manual operation, the response is lagging and easy to make mistakes, it is difficult to meet the demand of dynamic scheduling of resources in a distributed system. SUMMARY

[0005] The present application provides a database cluster computing task allocation method, device, equipment and medium, by performing secondary hash splitting when the pre-set hash shards cannot be evenly allocated to the computing nodes, solving the technical problem that the GBase 8a storage and computing separation cloud warehouse is limited in computing cluster specification, part of the node computing amount is tilted, the resource utilization is low, and cannot meet the user's flexible node scale demand in the pre-set HASH shard mode.

[0006] According to an aspect of the present application, a database cluster computing task allocation method is provided, the method comprising:

[0007] configuring the total number of hash shards, and determining the basic binding rules between the pre-set hash shards and the computing nodes;

[0008] obtaining the number of computing nodes corresponding to the current database cluster computing resources, and determining the judgment result of secondary splitting according to the number of computing nodes and the total number of hash shards;

[0009] When secondary splitting is needed, the number of equally divided shards is determined according to the number of computing nodes and the total number of hash shards, and a shard list is generated based on the number of equally divided shards, wherein the shard list includes an equally divided shard list and a secondary splitting shard list to be divided;

[0010] The computing task allocation is performed according to the shard list.

[0011] Optionally, the number of computing nodes corresponding to the computing resources of the current database cluster and the total number of preset hash shards are obtained to determine the judgment result of secondary splitting, including: determining the number of rows of a single hash shard; calculating a first ratio of the total number of hash shards to the number of computing nodes; when the first ratio is not an integer and the number of rows is greater than a preset row threshold, it is determined that secondary splitting is needed, otherwise, it is determined that secondary splitting is not needed.

[0012] Optionally, the number of equally divided shards is determined according to the number of computing nodes and the total number of hash shards, and a shard list is generated based on the number of equally divided shards, including: the first ratio is rounded down to obtain the number of equally divided shards; the shard number of each hash shard is obtained, and a second ratio of the shard number to the number of computing nodes is calculated; the hash shard with a second ratio less than or equal to the number of equally divided shards is stored in the equally divided shard list; and the hash shard with a second ratio greater than the number of equally divided shards is stored in the secondary splitting shard list.

[0013] Optionally, the computing task allocation is performed according to the shard list, including: when the shard list is the equally divided shard list, the computing task allocation is performed based on the basic binding rule.

[0014] Optionally, the computing task allocation is performed according to the shard list, including: when the shard list is the secondary splitting shard list, the list length of the secondary splitting shard list is determined, a third ratio of the total number of computing nodes to the list length is calculated; the parallelism of the query statement corresponding to the secondary splitting shard list is determined, and the number of secondary hash buckets is determined according to the product of the parallelism of the query statement and the third ratio; the original data corresponding to each hash shard in the secondary splitting shard list is obtained, the original data is divided based on the number of secondary hash buckets to generate each bucket number; the node number is calculated as the modulus result of the bucket number and the third ratio, and the node subscription relationship is determined based on the bucket number and the node number to complete the computing task allocation.

[0015] Optionally, the method further includes: when secondary splitting is not needed, a stable subscription relationship is directly generated according to the basic binding rule.

[0016] Optionally, the total number of hash shards is a multiple of 2.

[0017] According to another aspect of the present application, a database cluster computing task allocation device is provided, which comprises:

[0018] A parameter and rule configuration module is configured to configure the total number of hash shards and determine the basic binding rule between the preset hash shards and the computing nodes.

[0019] A secondary split result determination module is configured to obtain the number of computing nodes corresponding to the computing resources of the current database cluster and determine the judgment result of the secondary split according to the number of computing nodes and the total number of hash shards.

[0020] A shard list generation module is configured to determine the number of equally divided shards according to the number of computing nodes and the total number of hash shards when the secondary split is needed, and generate a shard list based on the number of equally divided shards, wherein the shard list includes an equally divided shard list and a list of shards to be secondarily split.

[0021] A computing task allocation module is configured to allocate computing tasks according to the shard list.

[0022] According to another aspect of the present application, an electronic device is provided, which comprises:

[0023] at least one processor;

[0024] and a memory connected in communication with the at least one processor;

[0025] wherein the memory stores a computer program capable of being executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the database cluster computing task allocation method according to any one of the embodiments of the present application.

[0026] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the database cluster computing task allocation method according to any one of the embodiments of the present application when executed.

[0027] The technical solution of the embodiments of the present application can expand the computing cluster specification adaptation range by configuring the total number of hash shards and determining the basic binding rule, and provide a basis for computing task allocation. It can accurately determine whether secondary split is needed, avoid redundant operations or computing tilt, determine the number of equally divided shards and generate a shard list when secondary split is needed, clearly define the number of shards stably divided by the nodes, and clearly distinguish different types of shards. According to the shard list, the computing task can be allocated, which can achieve efficient allocation without secondary split and balanced computing amount when secondary split is needed, and improve computing performance and resource utilization.

[0028] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative effort based on these drawings.

[0030] Figure 1 is a flow chart of a database cluster computing task allocation method according to an embodiment of the present application;

[0031] Figure 2 is a flow chart of another database cluster computing task allocation method according to an embodiment of the present application;

[0032] Figure 3 is a structural schematic diagram of a database cluster computing task allocation device according to an embodiment of the present application;

[0033] Figure 4 is a structural schematic diagram of an electronic device for implementing a database cluster computing task allocation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the person skilled in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should be within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment one

[0037] Figure 1A flowchart of a database cluster computing task allocation method is provided for the first embodiment of the application. The embodiment can be applied to a GBase 8a storage-computing separation cloud data warehouse computing scenario. The method can be executed by a database cluster computing task allocation device. The database cluster computing task allocation device can be realized in the form of hardware and / or software and can be configured in a computer controller. As shown in FIG. 8A, the method comprises the following steps. Figure 1

[0038] S110, configure the total number of hash segments and determine the basic binding rule between the preset hash segments and the computing nodes.

[0039] The total number of hash segments refers to the total number of preset hash segments configured by the read-only parameter_gbase_seg_count. The total number of hash segments is used to store the data in the database in a scattered manner and is the basis for dividing the data unit for computing task allocation. The default value of the total number of hash segments is 128, which can be flexibly configured according to the scale of the computing cluster nodes and the business requirements. The preset hash segment is a basic data segment unit divided according to the configured total number of hash segments when the database is initialized. In the GBase 8a cloud data warehouse, the data is first divided into 65536 HASH buckets by the HASH algorithm, and then the HASH buckets are allocated to different segments according to the number of preset hash segments. Each segment corresponds to a certain range of HASH bucket data and is the basic unit for data storage and computing task allocation. The computing node is a server node in the database cluster that undertakes data computing tasks. Multiple computing nodes form a computing cluster. Each computing node can correspond to one or more preset hash segments and obtain and process the computing tasks corresponding to the segments by establishing a subscription relationship with the segments. The basic binding rule is a rule for establishing a corresponding relationship between the preset hash segments and the computing nodes. Specifically, seg_id%node_count=node_id, where node_count is the number of computing cluster nodes and node_id is the computing node number, starting from 0. The basic binding rule is used to quickly determine the computing node corresponding to each segment without secondary segmentation, thereby guaranteeing the initial allocation order of the computing tasks.

[0040] Optionally, the total number of hash segments is a multiple of 2.

[0041] Specifically, the HASH segment number can be adjusted from an exponential of 2 to a multiple of 2 by adjusting the_gbase_seg_count parameter. For example, the HASH segment number is adjusted to 2x1x2x3x4x5=240. The preset HASH segments can be evenly allocated to the computing nodes in 20 specifications such as 1, 2, 3, 4, 5, 6, 8, 10, 12, 15, 16, 20, 24, 30, 40, 48, 60, 80, 120, and 240 nodes. ​

[0042] It should be noted that the multiple of 2 is selected as the total number of fragments because the multiple of 2 has more divisors and can form an integral relationship with more different numbers of computing nodes, that is, when the total number of hash fragments can be divided by the number of nodes, according to the basic binding rule, each computing node can be allocated the same number of hash fragments, for example, when 240 fragments correspond to 20 nodes, each node is allocated 12 fragments, avoiding the uneven distribution of fragments that leads to computing tilt in the initial allocation stage.

[0043] S120, obtain the number of computing nodes corresponding to the current database cluster computing resources, and determine the judgment result of the secondary split according to the number of computing nodes and the total number of hash fragments.

[0044] Among them, the database cluster computing resources refer to the set of computing nodes in the current cluster that can be used to process computing tasks, and the core index is the number of computing nodes, which directly affects the allocation mode and processing efficiency of computing tasks. Secondary split refers to the operation of re-splitting the hash fragments that cannot be evenly distributed when the total number of pre-set hash fragments cannot be evenly distributed to the computing nodes, the purpose is to realize the balance of the computing amount of each computing node, and avoid the occurrence of computing tilt, without creating a temporary table for HASH redistribution.

[0045] Optionally, the number of computing nodes corresponding to the current database cluster computing resources and the total number of pre-set hash fragments are obtained to determine the judgment result of the secondary split, including: determining the number of rows in a single hash fragment; calculating the first ratio of the total number of hash fragments and the number of computing nodes, and when the first ratio is not an integer and the number of rows is greater than a pre-set row threshold, it is determined that secondary split is needed, otherwise, it is determined that secondary split is not needed.

[0046] It can be known that since multiple HASH buckets in the GBase 8a cloud warehouse are stored in the same data unit DataCell, multiple DataCells form an aggregated storage file PCFile, and each pre-set hash fragment corresponds to a plurality of PCFiles, the system will count the total number of data rows contained in each pre-set hash fragment, that is, the total number of record rows stored in all PCFiles under the fragment.

[0047] Specifically, the system will calculate the first ratio of the total number of hash fragments and the number of computing nodes, and determine whether it is an integer to determine whether the pre-set hash fragment can be evenly distributed to each computing node by the basic binding rule. If the ratio is an integer, it means that each computing node can be allocated the same number of pre-set hash fragments, and at this time the fragments can be evenly distributed and do not need to be secondary split. If the ratio is not an integer, it means that the pre-set hash fragment cannot be evenly distributed by the basic rule, and there is a situation that some nodes are allocated more fragments than other nodes, which may lead to computing tilt, and further combined with the number of rows in the fragment to determine whether to trigger secondary split.

[0048] The preset row threshold can be 65536 rows, and the setting of the threshold matches the HASH bucket split radix of the GBase 8a cloud number warehouse, so that only the shard with a large enough data quantity has the necessity of secondary split. Therefore, when the two conditions are simultaneously satisfied, that is, the first ratio is not an integer, and the number of rows of the hash shard currently judged is greater than 65536 rows, the system determines that secondary split is needed, because the shard cannot be evenly distributed to the nodes at this time, and has enough data quantity to support secondary split to realize balanced calculation. If either of the two conditions is not satisfied, that is, the first ratio is an integer, or the number of rows of the shard is less than or equal to 65536 rows, the system determines that secondary split is not needed, and at this time, the subscription relationship between the shard and the computing node is established according to the basic binding rule, and the computing task corresponding to the shard is allocated to the corresponding node.

[0049] S130, when secondary split is needed, determining the equal division shard number according to the number of computing nodes and the total number of hash shards, and generating a shard list based on the equal division shard number, wherein the shard list includes an equal division shard list and a to-be-secondary-split shard list.

[0050] The equal division shard number refers to the number of preset hash shards that can be evenly distributed to each computing node. The shard list is a shard classification set divided according to the equal division shard number, and includes the equal division shard list and the to-be-secondary-split shard list.

[0051] Optionally, the equal division shard number is determined according to the number of computing nodes and the total number of hash shards, and the shard list is generated based on the equal division shard number, including: taking the first ratio down to obtain the equal division shard number; obtaining the shard number of each hash shard, and calculating the second ratio of the shard number and the number of computing nodes; storing the hash shard with the second ratio less than or equal to the equal division shard number in the equal division shard list; and storing the hash shard with the second ratio greater than the equal division shard number in the to-be-secondary-split shard list.

[0052] The equal division shard number is the number of preset hash shards that can be evenly distributed to each computing node. The system takes the first ratio of the total number of hash shards and the number of computing nodes down to obtain the equal division shard number. By taking the first ratio down, it can be ensured that the equal division shard number obtained is the lower limit of the number of shards that can be stably distributed to all computing nodes, avoiding the chaos of the number of distributed shards due to the inability to divide, and providing a unified standard for subsequent shard classification.

[0053] Then the shard list is generated based on the equal division number of shards, which needs to be classified according to the ratio of shard number to the number of computing nodes, and the specific steps are carried out around the shard number. The system will calculate the second ratio of the shard number seg_i to the number of computing nodes node_count, that is, seg_id / node_count. The second ratio reflects the position of the shard in the overall allocation sequence, and can accurately distinguish between the evenly divisible basic shards and the special shards that need to be processed twice, providing clear data classification basis for the two paths of subsequent computing task allocation. The system will store the hash shards with a second ratio less than or equal to the equal division number of shards in the equal division shard list. And the hash shards with a second ratio greater than the equal division number of shards are stored in the list of shards to be processed twice.

[0054] Optionally, the method further comprises: when no secondary splitting is needed, generating a stable subscription relationship directly according to the basic binding rule.

[0055] Wherein, the basic binding rule is: seg_id% node_count = node_id, wherein seg_id is the unique number of each preset hash shard, node_count is the total number of nodes of the current computing cluster, and node_id is the unique number of the computing node.

[0056] Specifically, when no secondary splitting is needed, the system will traverse all preset hash shards, and perform seg_id% node_count operation on the seg_id of each shard. The result obtained is the node_id corresponding to the shard, and then the computing task of the shard is determined to be undertaken by the computing node corresponding to the node_id. For example, when _gbase_seg_count=128 and node_count=32, the shard with seg_id=0 calculates 0%32=0, which corresponds to the node with node_id=0; the shard with seg_id=32 calculates 32%32=0, which also corresponds to the node with node_id=0. Finally, each node will correspond to 4 shards, such as node_id=0 corresponding to seg_id=0, 32, 64, and 96, forming a fixed subscription relationship.

[0057] S140, computing task allocation according to the shard list.

[0058] Wherein, the computing task allocation is the process of distributing the computing task corresponding to each shard to the corresponding computing node according to the subscription relationship between the shard and the computing node.

[0059] Optionally, the computing task allocation according to the shard list comprises: when the shard list is the equal division shard list, the computing task allocation is based on the basic binding rule.

[0060] Specifically, when the slice list is the uniform slice list, the system traverses all slice numbers seg_id in the uniform slice list, and performs seg_id%node_count operation on each seg_id one by one: the operation result is the computing node number node_id corresponding to the slice, and then it is determined that the computing task borne by the slice is undertaken by the computing node corresponding to the node_id. After the allocation is completed, each computing node can directly read the PCFile data corresponding to the subscribed uniform slice, start multi-thread parallel computing, and does not need to perform additional data redistribution or secondary splitting, thereby guaranteeing the efficiency and load balancing of the computing task allocation.

[0061] The technical scheme of the embodiment of the application can expand the computing cluster specification adaptation range by configuring the total number of hash slices and determining the basic binding rule, and provide a basis for computing task allocation. Whether secondary splitting is needed can be accurately defined, and redundant operations or computing tilt can be avoided. When secondary splitting is needed, the number of uniform slices is determined and a slice list is generated, the number of stably divided slices of the node can be determined, and different types of slices can be clearly distinguished. According to the slice list, the computing task can be allocated, and the efficient allocation without secondary splitting and the balanced computing amount with secondary splitting can be realized, thereby improving the computing performance and resource utilization.

[0062] Embodiment Two

[0063] Figure 2 A flowchart of a database cluster computing task allocation method provided by the embodiment two of the application is provided, and the embodiment two adds a specific process of computing task allocation according to the slice list when the slice list is the to-be-secondary-split slice list on the basis of the embodiment one. The specific content of steps S210-S230 is substantially the same as that of steps S110-S130 in the embodiment one, and therefore the embodiment two will not be described in detail. As shown in the embodiment two, the method comprises the following steps. Figure 2

[0064] S210, configure the total number of hash slices, and determine the basic binding rule between the preset hash slice and the computing node.

[0065] Optionally, the total number of hash slices is a multiple of 2.

[0066] S220, obtain the number of computing nodes corresponding to the current database cluster computing resource, and determine the secondary splitting judgment result according to the number of computing nodes and the total number of hash slices.

[0067] ​Optionally, the number of computing nodes corresponding to the computing resources of the current database cluster and the total number of preset hash shards are acquired to determine the judgment result of the secondary splitting, including: determining the number of rows of a single hash shard; calculating a first ratio of the total number of hash shards to the number of computing nodes; when the first ratio is not an integer and the number of rows is greater than a preset threshold of the number of rows, it is determined that secondary splitting is needed, otherwise, it is determined that secondary splitting is not needed.

[0068] S230, when secondary splitting is needed, the number of evenly divided shards is determined according to the number of computing nodes and the total number of hash shards, and a shard list is generated based on the number of evenly divided shards, wherein the shard list includes an evenly divided shard list and a list of shards to be secondary split.

[0069] Optionally, the number of evenly divided shards is determined according to the number of computing nodes and the total number of hash shards, and a shard list is generated based on the number of evenly divided shards, including: rounding down the first ratio to obtain the number of evenly divided shards; obtaining the shard number of each hash shard, and calculating a second ratio of the shard number to the number of computing nodes; storing the hash shard with a second ratio less than or equal to the number of evenly divided shards in the evenly divided shard list; and storing the hash shard with a second ratio greater than the number of evenly divided shards in the list of shards to be secondary split.

[0070] Optionally, the method further comprises: when secondary splitting is not needed, generating a stable subscription relationship directly according to the basic binding rule.

[0071] S240, when the shard list is the list of shards to be secondary split, the list length of the list of shards to be secondary split is determined, and a third ratio of the total number of computing nodes to the list length is calculated.

[0072] Specifically, when the shard list is the list of shards to be secondary split, the system will obtain the list length of the list of shards to be secondary split, that is, the total number of preset hash shards contained in the list, denoted as seg_list.size, for example, if 10 shards cannot be evenly divided by the basic rule, seg_list.size is 10; then, the system will obtain the total number of computing nodes node_count of the current computing cluster, such as 30 nodes, and calculate the third ratio of the two, that is, node_count / seg_list.size, which represents how many computing nodes each shard to be secondary split needs to be subscribed to, for example, when node_count=30, seg_list.size=10, the third ratio is 3, which means that each shard to be secondary split needs to be processed by 3 nodes.

[0073] S250, determining the parallelism of the query statement corresponding to the list of shards to be secondary split, and determining the number of secondary hash buckets according to the product of the parallelism of the query statement and the third ratio.

[0074] Specifically, the system determines the parallelism degree of the query statement corresponding to the list of secondary split shards, i.e., the number of parallel processing threads that can be started on a single computing node for the SQL query task. Then, the product of the parallelism degree of the query statement and the third ratio is calculated, and the smaller value of the product and 128 is taken as the number of secondary hash buckets, for example, when degree = 4 and the third ratio = 3, the product is 12, which is less than 128, so the number of secondary hash buckets is 12. If degree = 40 and the third ratio = 4, the product is 160, which is greater than 128, so the number of buckets is 128. In summary, the product of the parallelism degree and the number of shared nodes is used to determine the bucketing base, while the maximum number of buckets is limited, which not only ensures the efficiency of multi-node and multi-thread parallel processing, but also avoids resource waste caused by excessive bucketing.

[0075] S260, obtain the original data corresponding to each hash shard in the list of secondary split shards, and split the original data based on the number of secondary hash buckets to generate each bucket number.

[0076] Specifically, the system will split the data of the secondary split shard according to the bucketing rule. First, the original data corresponding to each hash shard in the list of secondary split shards is obtained, and the original data is stored in the PCFile corresponding to the shard, which contains the specific records of all HASH buckets under the shard. Then, based on the determined number of secondary hash buckets, the original data is secondary HASH split, the existing HASH split algorithm of GBase 8a cloud warehouse is reused, the split dimension is adjusted, the original data of the same secondary split shard is split into a corresponding number of new HASH buckets, i.e., secondary hash buckets, and each new bucket is assigned a unique bucket number, denoted as hash2id, to ensure that the data volume of each bucket is relatively balanced.

[0077] S270, calculate the modulo result of the bucket number and the third ratio as the node number, and determine the node subscription relationship based on the bucket number and the node number to complete the calculation task allocation.

[0078] Specifically, the system calculates the modulo result of the hash2id ratio of each bucket number to the third value. This result is the node_id of the computing node responsible for processing the data in that bucket. For example, when the third ratio = 3 and hash2id = 0, 0%3 = 0, corresponding to node_id = 0. This ensures that the different buckets of each shard to be further split are evenly distributed among their shared nodes. Then, based on the correspondence between bucket numbers and node numbers, the system generates a subscription relationship between each computing node and the buckets. That is, each node only subscribes to the bucket data that matches its node_id. Each node reads the secondary split shard data corresponding to its own node, completes the secondary split, and only performs calculations on the bucket data that matches its node_id; other data is ignored (and calculated by other matching nodes). Finally, each computing node starts a separate thread to process the computation tasks of the subscribed buckets and merges the calculation results with the results of other buckets that do not require secondary splitting, completing the overall computation task allocation. The entire process does not require creating temporary tables for hash redistribution and only needs to read local bucket data, which greatly reduces redundant IO reads. At the same time, it solves the computational skew problem caused by uneven distribution of fragments to be split in the secondary partitioning, and achieves the optimal balance between computational load and resource utilization.

[0079] The technical solution of this invention provides a basic ratio for subsequent bucketing and node binding by specifying the number of nodes that each shard to be secondary-sharded needs to be subscribed to. Determining the number of secondary hash buckets matches the processing needs of multiple nodes and multiple threads, while avoiding resource waste caused by too many buckets. By splitting the original data based on the number of secondary hash buckets and generating bucket numbers, the shard data to be secondary-sharded can be divided into balanced local bucket data. By establishing node subscription relationships, the even distribution of the shard data to be secondary-sharded across nodes is achieved, completing the balanced scheduling of computing tasks.

[0080] Example 3

[0081] Figure 3 This is a schematic diagram of a database cluster computing task allocation device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a parameter and rule configuration module 310, used to configure the total number of hash shards and determine the basic binding rules between preset hash shards and computing nodes;

[0082] The secondary partitioning result determination module 320 is used to obtain the number of computing nodes corresponding to the current database cluster computing resources, and determine the judgment result of the secondary partitioning based on the number of computing nodes and the total number of hash shards;

[0083] The slice list generation module 330 is configured to determine an equal division slice number according to the number of computing nodes and the total number of hash slices when secondary division is needed, and generate a slice list based on the equal division slice number, wherein the slice list comprises an equal division slice list and a secondary division slice list.

[0084] The computing task allocation module 340 is configured to perform computing task allocation according to the slice list.

[0085] Optionally, the secondary division result determination module 320 is specifically configured to determine the number of rows of a single hash slice, calculate a first ratio of the total number of hash slices to the number of computing nodes, and determine that secondary division is needed when the first ratio is not an integer and the number of rows is greater than a preset row threshold, or determine that secondary division is not needed.

[0086] Optionally, the slice list generation module 330 is specifically configured to perform floor operation on the first ratio to obtain the equal division slice number, obtain the slice number of each hash slice, calculate a second ratio of the slice number to the number of computing nodes, and store the hash slice with the second ratio less than or equal to the equal division slice number in the equal division slice list, and store the hash slice with the second ratio greater than the equal division slice number in the secondary division slice list.

[0087] Optionally, the computing task allocation module 340 comprises an equal division slice list allocation unit configured to perform computing task allocation based on a basic binding rule when the slice list is the equal division slice list.

[0088] Optionally, the computing task allocation module 340 comprises a secondary division slice list allocation unit configured to determine the list length of the secondary division slice list when the slice list is the secondary division slice list, calculate a third ratio of the total number of computing nodes to the list length, determine the parallelism of a query statement corresponding to the secondary division slice list, determine the number of secondary hash buckets according to the product of the parallelism of the query statement and the third ratio, obtain the original data corresponding to each hash slice in the secondary division slice list, perform division on the original data based on the number of secondary hash buckets to generate each bucket number, calculate the modulo result of the bucket number and the third ratio as a node number, and determine a node subscription relationship based on the bucket number and the node number to complete computing task allocation.

[0089] Optionally, the apparatus further comprises a secondary division free module configured to directly generate a stable subscription relationship according to a basic binding rule when secondary division is not needed.

[0090] The technical scheme of the embodiment of the present application can expand the computing cluster specification adaptation range by configuring the total number of hash fragments and determining the basic binding rule, and provide a basis for computing task allocation. Whether secondary splitting is needed can be accurately defined, and redundant operations or computing tilt can be avoided. When secondary splitting is needed, the equal division fragment number is determined and the fragment list is generated, the number of stably divided fragments of the node can be clearly defined, and different types of fragments can be clearly distinguished. According to the fragment list, the computing task can be allocated, the efficient allocation when no secondary splitting is needed and the balanced computing amount when secondary splitting is needed can be realized, and the computing performance and resource utilization rate are improved.

[0091] The database cluster computing task allocation device provided by the embodiment of the present application can execute the database cluster computing task allocation method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0092] Embodiment four

[0093] Figure 4 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0094] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected with the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected with each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0096] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as a database cluster computing task allocation method.

[0097] In some embodiments, a database cluster computing task allocation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of a database cluster computing task allocation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform a database cluster computing task allocation method by any other appropriate means, such as by means of firmware.

[0098] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0099] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0100] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0102] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0103] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0104] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved, and the present disclosure is not limited herein.

[0105] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principles of the present disclosure should be included in the scope of the present disclosure.

Claims

1. A database cluster computing task allocation method, characterized in that, The method comprises the following steps: configuring the total number of hash shards and determining the basic binding rule between the preset hash shards and the computing nodes; obtaining the number of computing nodes corresponding to the computing resources of the current database cluster, and determining the judgment result of secondary splitting according to the number of computing nodes and the total number of hash shards; when secondary splitting is needed, determining the number of equally divided shards according to the number of computing nodes and the total number of hash shards, and generating a shard list based on the number of equally divided shards, wherein the shard list comprises an equally divided shard list and a to-be-secondary-split shard list; performing computing task allocation according to the shard list; wherein, the number of computing nodes corresponding to the computing resources of the current database cluster and the total number of preset hash shards are obtained to determine the judgment result of secondary splitting, comprising: determining the number of rows in a single hash shard; calculating the first ratio of the total number of hash shards and the number of computing nodes, and when the first ratio is not an integer and the number of rows is greater than a preset row threshold, it is determined that secondary splitting is needed, otherwise, it is determined that secondary splitting is not needed.

2. The method of claim 1, wherein, determining the number of equally divided shards by rounding down the first ratio, and obtaining the shard number of each hash shard and the second ratio of the number of computing nodes; storing the hash shards with the second ratio less than or equal to the number of equally divided shards in the equally divided shard list; storing the hash shards with the second ratio greater than the number of equally divided shards in the to-be-secondary-split shard list. when the shard list is the equally divided shard list, performing computing task allocation based on the basic binding rule. when the shard list is the to-be-secondary-split shard list, determining the list length of the to-be-secondary-split shard list, calculating the third ratio of the total number of computing nodes and the list length; 3. The method of claim 2, wherein, determining the query statement parallelism corresponding to the to-be-secondary-split shard list, and determining the number of secondary hash buckets according to the product of the query statement parallelism and the third ratio; obtaining the original data corresponding to each hash shard in the to-be-secondary-split shard list, and performing splitting on the original data based on the number of secondary hash buckets to generate each bucket number; 4. The method of claim 2, wherein, calculating the modulo result of the bucket number and the third ratio as the node number, and determining the node subscription relationship based on the bucket number and the node number to complete the computing task allocation. The method further comprises: when secondary splitting is not needed, directly generating a stable subscription relationship according to the basic binding rule. The total number of hash shards is a multiple of 2. The method comprises the following steps:

5. The method of claim 1, wherein, a parameter and rule configuration module for configuring the total number of hash shards and determining the basic binding rule between the preset hash shards and the computing nodes; a secondary splitting result determination module for obtaining the number of computing nodes corresponding to the computing resources of the current database cluster, and determining the judgment result of secondary splitting according to the number of computing nodes and the total number of hash shards; 6. The method of claim 1, wherein, ​ 7. A database cluster computing task allocation apparatus characterized by comprising: ​ ​ ​ The slice list generation module is configured to determine an equal division slice number according to the number of computing nodes and the total number of hash slices when secondary division is needed, and generate a slice list based on the equal division slice number, wherein the slice list comprises an equal division slice list and a list of slices to be secondarily divided. The computing task allocation module is configured to perform computing task allocation according to the slice list. The secondary division result determination module is configured to determine the number of rows of a single hash slice. The first ratio of the total number of hash slices to the number of computing nodes is calculated, and when the first ratio is not an integer and the number of rows is greater than a preset row threshold, it is determined that secondary division is needed, otherwise, it is determined that secondary division is not needed.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program capable of being executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-6.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions for enabling the processor to implement the method of any one of claims 1-6 when executed.

Images