Database partition capacity expansion and access method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide a database partition capacity expansion and access method and device, a storage medium and a program product. The database partition capacity expansion method comprises the steps that the total number of data records in a current partition is obtained; under the condition that the total number is greater than a partition capacity threshold value, acquiring newly added nodes to obtain a node set; dividing the nodes in the node set into a first type of newly arranged nodes and a second type of newly arranged nodes; determining a sub-partition migration plan for a plurality of sub-partitions in the current partition, wherein the sub-partition migration plan comprises migrating a first part of sub-partitions to a first new partition and migrating a second part of sub-partitions to a second new partition; and triggering data record migration of the new partition, so that an original node in the first type of newly-arranged nodes migrates the data record in the first part of sub-partition to a newly-added node in the first type of newly-arranged nodes, and an original node in the second type of newly-arranged nodes migrates the data record in the second part of sub-partition to a newly-added node in the second type of newly-arranged nodes.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a database partition expansion and access method, device, storage medium, and program product. Background Art

[0002] Database partitioning technology can divide large database tables or indexes into multiple smaller and easier-to-manage parts, called partitions. This partitioning technology can improve database query performance, improve system resource utilization, and simplify database maintenance, which has many benefits. However, as business needs change, the amount of data may continue to grow, and the original partitioning method will be difficult to maintain efficient performance. Summary of the invention

[0003] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a method for expanding database partition is proposed, wherein each data record in the partition belongs to a sub-partition, and the method is applied to a partition management component, comprising:

[0005] Obtain the total number of data records in the current partition, where the current partition includes multiple sub-partitions;

[0006] When the total number is greater than the partition capacity threshold, a number of newly added nodes are acquired to obtain a node set, where the node set includes the number of newly added nodes and a number of original nodes that maintain the current partition;

[0007] Dividing the nodes in the node set into a first type of newly established nodes and a second type of newly established nodes, wherein the first type of newly established nodes maintains a first new partition, and the second type of newly established nodes maintains a second new partition, and both the first type of newly established nodes and the second type of newly established nodes include both the original nodes and the newly added nodes;

[0008] Determine a subpartition migration plan for the multiple subpartitions in the current partition, the subpartition migration plan comprising migrating a first portion of the multiple subpartitions to the first new partition and migrating a second portion of the multiple subpartitions to the second new partition;

[0009] Trigger the migration of new partition data records so that the original nodes in the first type of newly established nodes migrate the data records in the first part of the sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of the sub-partitions to the newly added nodes in the second type of newly established nodes.

[0010] According to a second aspect of one or more embodiments of this specification, a database access method is proposed, which is applied to a routing component, including:

[0011] receiving a database access request, wherein the database access request carries information of a target data record;

[0012] Query the target sub-partition to which the target data record belongs according to the information of the target data record;

[0013] According to the mapping relationship between sub-partitions and partitions, query the target partition corresponding to the target sub-partition;

[0014] Sending the database access request to a database node maintaining the target partition;

[0015] The target partition is expanded using the aforementioned method.

[0016] According to a third aspect of one or more embodiments of the present specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the aforementioned method by running the executable instructions.

[0017] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the aforementioned method are implemented.

[0018] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is proposed, comprising a computer program / instruction, which implements the steps of the aforementioned method when executed by a processor.

[0019] As can be seen from the above description, this manual can divide several newly added nodes and several original nodes maintaining the current partition into a first type of newly established nodes maintaining the first new partition and a second type of newly established nodes maintaining the second new partition when the total number of data records in the current partition is greater than the partition capacity threshold, and determine the first part of sub-partitions to be migrated to the first new partition and the second part of sub-partitions to be migrated to the second new partition among the multiple sub-partitions included in the current partition, and then trigger data record migration so that the original nodes in the first type of newly established nodes migrate the data records in the first part of sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of sub-partitions to the newly added nodes in the second type of newly established nodes, thereby migrating the data records in the current partition to the first new partition and the second new partition in units of sub-partitions, thereby realizing automatic expansion of the partition.

[0020] It can be seen that this manual introduces the concept of sub-partitions, which can logically divide the data records in the partition into sub-partitions. When the partition is subsequently expanded, the data records are migrated using sub-partitions as migration units, thereby achieving more flexible and convenient partition expansion, thereby better adapting to the growth of data volume and changes in business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The diagram is a schematic diagram of the architecture of a database partition capacity expansion system provided by an exemplary embodiment.

[0022] Figure 2 The present invention is a flowchart of a method for expanding database partitions provided by an exemplary embodiment.

[0023] Figure 3 The present invention is a flowchart of another database partition expansion method provided by an exemplary embodiment.

[0024] Figure 4 It is a schematic diagram of a database node partition expansion division provided by an exemplary embodiment.

[0025] Figure 5 It is a flowchart of a database access method provided by an exemplary embodiment.

[0026] Figure 6 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0027] Figure 7 It is a block diagram of a database partition expansion device provided by an exemplary embodiment.

[0028] Figure 8 It is a block diagram of a database access device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0029] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] Database partitioning technology can divide large database tables or indexes into multiple smaller and easier-to-manage parts, called partitions. This partitioning technology can improve database query performance, improve system resource utilization, and simplify database maintenance, which has many benefits. However, as business needs change, the amount of data may continue to grow, and the original partitioning method will be difficult to maintain efficient performance.

[0031] This manual provides a database partition expansion solution that can automatically increase the number of partitions based on the amount of data, achieving horizontal expansion of the database to meet the growth of data volume and changes in business needs.

[0032] Figure 1 FIG. 1 is a schematic diagram of the architecture of a database partition expansion system provided by an exemplary embodiment. Figure 1 As shown, the system may include a partition management component 1011 and several database nodes, such as database node 1021, database node 1022, database node 1023, ..., database node 102N, etc.

[0033] Among them, the database node can be regarded as a process running a database service, which can be used to maintain partitions, store and manage data records, and is usually a single physical or virtual server.

[0034] The partition management component 1011 is a database system-level component used to maintain partition metadata. It can be responsible for the creation and management of partitions, maintenance of partition information, partition balancing strategies, etc.

[0035] Figure 2 The present invention is a flowchart of a method for expanding database partitions provided by an exemplary embodiment.

[0036] In this embodiment, the concept of sub-partition is proposed. Each data record in a database partition belongs to a sub-partition, and the data records in each database partition usually belong to multiple sub-partitions.

[0037]

[0038] Table 1

[0039] Please refer to the example in Table 1. Assume that database partition A includes 400 data records, of which data records 001-100 belong to subpartition 0001, data records 101-200 belong to subpartition 0002, data records 201-300 belong to subpartition 0003, and data records 301-400 belong to subpartition 0004. In other words, database partition A includes four subpartitions 0001-0004.

[0040] It should be noted that Table 1 is only for illustrative purposes. In an actual database instance, the number of data records in each database partition may be as high as millions.

[0041] In this embodiment, the number of data records that each subpartition can accommodate can be pre-planned based on business needs, which is called the unit number, and then the corresponding subpartition can be allocated to the data record when it is created. For example, assuming that the unit number of the subpartition is 100, for database partition A, the 001th data record created therein can be allocated to subpartition 0001, the 002th data record created can be allocated to subpartition 0001, and the 100th data record can also be allocated to subpartition 0001. When the 101st data record is created, since the number of data records in subpartition 0001 has reached the unit number of 100, subpartition 0002 will be allocated to the 101st data record, and then the 102nd data record will also be allocated to subpartition 0002, and so on.

[0042] In this embodiment, the partition management component can maintain the mapping relationship between the sub-partitions and the partitions, and the mapping relationship can be used to guide routing. The specific implementation method will be described in detail in the subsequent embodiments. Still taking Table 1 as an example, the partition management component will record the mapping relationship between sub-partitions 0001-0004 and database partition A, and the mapping relationship can be stored after the data record is assigned to the new sub-partition.

[0043] In this embodiment, more flexible partition expansion can be achieved based on the sub-partitions. Figure 2 The database partition expansion method of this embodiment can be applied to the partition management component, and includes the following steps:

[0044] Step 202: Obtain the total number of data records in the current partition, where the current partition includes multiple sub-partitions.

[0045] In this embodiment, before performing the data recording operation, the total number of data records in the partition to which the data record belongs may be obtained. For ease of description, the partition to which the data record belongs may be referred to as the current partition.

[0046] For example, before creating a new data record, you can obtain the total number of data records in the current partition to which the data record belongs.

[0047] In this embodiment, the total number of data records in the current partition may be obtained from the partition metadata.

[0048] Step 204, when the total number is greater than the partition capacity threshold, obtain a number of newly added nodes to obtain a node set, wherein the node set includes the number of newly added nodes and a number of original nodes that maintain the current partition.

[0049] Based on the aforementioned step 202, after the total number of data records in the current partition is obtained, the size relationship between the total number and the partition capacity threshold can be compared. The partition capacity threshold can be a preset upper limit value of the number of data records that can ensure the partition query performance. Assuming that the partition capacity threshold of the current partition is 400, then when the number of data records in the current partition is less than or equal to 400, the query performance of the current partition is better. If the number of data records in the current partition is greater than 400, then as the number of data records gradually increases, the query performance of the current partition will gradually decrease. Among them, the acquisition and comparison of the total number of data in the current partition can be achieved through a stored procedure trigger.

[0050] In this embodiment, if it is determined through comparison that the total number of data records in the current partition is greater than the partition capacity threshold, partition expansion may be triggered to alleviate the pressure on the current partition.

[0051] When performing partition expansion, a database node for partition expansion may be obtained first, which is referred to as a newly added node. In contrast, a database node for maintaining the current partition may be referred to as an original node. The original node and the newly added node may form a node set.

[0052] In this embodiment, considering that Byzantine nodes may appear in the database system and threaten the security of the database, the PBFT (Practical Byzantine Fault Tolerance) consensus protocol can be used to set multiple original nodes for each database partition, and these original nodes jointly maintain the database partition. Specifically, the PBFT protocol can tolerate up to f Byzantine nodes in a system with 3f+1 nodes. If f=1, 4 original nodes can be configured for each database partition. Among these 4 original nodes, 1 original node maintains the master copy of the corresponding database partition, and the remaining 3 original nodes maintain the slave copies of the corresponding database partition.

[0053] In this embodiment, taking 4 original nodes as partition units to maintain a database partition as an example, when the partition is expanded, 4 new nodes can be obtained, so the obtained node set includes 8 database nodes, namely 4 original nodes and 4 new nodes.

[0054] Of course, if other consensus protocols are used, when expanding the partition, the same number of new nodes as the original nodes can be obtained, or a different number of new nodes can be obtained. This manual does not impose any special restrictions on this.

[0055] Step 206, divide the nodes in the node set into a first type of newly established nodes and a second type of newly established nodes, the first type of newly established nodes maintains a first new partition, the second type of newly established nodes maintains a second new partition, and the first type of newly established nodes and the second type of newly established nodes both include the original nodes and the newly added nodes.

[0056] Based on the aforementioned step 204, after acquiring the newly added nodes to obtain the node set, the nodes in the node set can be divided into a first type of newly established nodes and a second type of newly established nodes. The first type of newly established nodes are used to maintain the first new partition, and the second type of newly established nodes are used to maintain the second new partition. The first new partition and the second new partition are two different new partitions obtained after the current partition is partitioned and expanded.

[0057] In one example, nodes may be selected one by one from the node set, and the selected nodes may be original nodes or newly added nodes. The selected nodes are then sequentially determined as first-category newly established nodes and second-category newly established nodes to achieve the division of each node in the node set.

[0058] For example, assuming that the node set includes 8 nodes, the node selected from the node set for the first time can be determined as a first-type newly established node, the node selected from the node set for the second time can be determined as a second-type newly established node, the node selected from the node set for the third time can be determined as a first-type newly established node, the node selected from the node set for the fourth time can be determined as a second-type newly established node, and so on.

[0059] After all nodes in the node set have been divided, for the first type of newly established nodes, it can be checked whether the first type of newly established nodes includes both original nodes and newly added nodes. At the same time, for the second type of newly established nodes, it can be checked whether the second type of newly established nodes also includes both original nodes and newly added nodes. If both are true, the subsequent steps can be continued. If there are only original nodes or only newly added nodes in the first type of newly established nodes, or there are only original nodes or only newly added nodes in the second type of newly established nodes, the nodes in the node set can be divided again.

[0060] Among them, if the number of original nodes and the number of newly added nodes in the node set are the same, when it is determined that the first type of newly established nodes includes both original nodes and newly added nodes, it can be determined that the second type of newly established nodes will also include both original nodes and newly added nodes, and subsequent steps can be continued.

[0061] In another example, considering that there may be Byzantine nodes in both the original nodes and the newly added nodes, a distributed random algorithm can be used to divide the nodes in the node set to ensure that two Byzantine nodes are not divided into the same type of newly established nodes.

[0062] For example, a unique number may be assigned to all nodes in a node set, and then a distributed random algorithm may be used to generate a random number sequence, each number in the random number sequence may correspond to a node number, and then all nodes may be sorted according to the generated random number sequence so that each node is arranged in a random order, and then the sorted nodes may be divided into two categories of newly established nodes.

[0063] Similarly, after the division is completed, the first type of newly established nodes and the second type of newly established nodes may also be verified. The verification method may refer to the previous description.

[0064] In this example, a distributed random algorithm is used to divide the nodes in the node set into the first category of newly established nodes and the second category of newly established nodes. When there are two Byzantine nodes in the node set, it can ensure that the two Byzantine nodes are divided into different categories of newly established nodes, thereby ensuring the security of the database.

[0065] Step 208: determine a subpartition migration plan for the multiple subpartitions in the current partition, wherein the subpartition migration plan includes migrating a first portion of the multiple subpartitions to the first new partition and migrating a second portion of the multiple subpartitions to the second new partition.

[0066] In this embodiment, the data records in the current partition need to be migrated to the first new partition and the second new partition. To achieve a better partition expansion effect, half of the data records in the current partition can be migrated to the first new partition as much as possible, and the remaining half of the data records can be migrated to the second new partition, so that the number of data records in the first new partition and the second new partition is as equal as possible after the data records are migrated.

[0067] In this embodiment, when migrating data records with sub-partitions as migration units, a sub-partition migration plan for the current partition can be determined first, and the sub-partition migration plan includes migrating part of the multiple sub-partitions (hereinafter referred to as the first part of sub-partitions) to the first new partition and migrating another part of the multiple sub-partitions (hereinafter referred to as the second part of sub-partitions) to the second new partition.

[0068] For example, assuming that the current partition includes 4 sub-partitions, when performing data migration, the data records in the first two sub-partitions can be migrated to the first new partition, and the data records in the remaining two sub-partitions can be migrated to the second new partition, that is, the first two sub-partitions are the first part of the sub-partitions, and the remaining two sub-partitions are the second part of the sub-partitions. The migration plan is to migrate the first two sub-partitions to the first new partition, and migrate the last two sub-partitions to the second new partition. If the current partition includes 5 sub-partitions, when performing data migration, the data records in the first 3 sub-partitions can be migrated to the first new partition, and the data records in the remaining two sub-partitions can be migrated to the second new partition, that is, the first 3 sub-partitions are the first part of the sub-partitions, and the remaining two sub-partitions are the second part of the sub-partitions.

[0069] Step 210 triggers the migration of new partition data records so that the original nodes in the first type of newly established nodes migrate the data records in the first part of the sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of the sub-partitions to the newly added nodes in the second type of newly established nodes.

[0070] Based on the aforementioned step 208, after determining the first part of sub-partitions that need to be migrated to the first new partition and the second part of sub-partitions that need to be migrated to the second new partition, data record migration can be performed.

[0071] In this embodiment, since each original node includes all data records in the current partition, when performing data migration, for the original node, it is sufficient to delete the data records that no longer need to be maintained in the new partition to which it belongs, that is, to delete the data records maintained by another new partition.

[0072] Specifically, the original nodes in the first type of newly established nodes migrate the data records in the first part of the sub-partitions to each newly added node in the first type of newly established nodes, and delete the data records in the second part of the sub-partitions, that is, delete the data records in the sub-partitions migrated to the second new partition. Similarly, the original nodes in the second type of newly established nodes migrate the data records in the second part of the sub-partitions to each newly added node in the second type of newly established nodes, and delete the data records in the first part of the sub-partitions, that is, delete the data records in the sub-partitions migrated to the first new partition.

[0073] When migrating data records within a new partition, the newly added node may initiate a record migration request to the original node, or the original node may actively initiate record migration, and this specification does not impose any special restrictions on this.

[0074] As can be seen from the above description, this manual can divide several newly added nodes and several original nodes maintaining the current partition into a first type of newly established nodes maintaining the first new partition and a second type of newly established nodes maintaining the second new partition when the total number of data records in the current partition is greater than the partition capacity threshold, and determine the first part of sub-partitions to be migrated to the first new partition and the second part of sub-partitions to be migrated to the second new partition among the multiple sub-partitions included in the current partition, and then trigger data record migration so that the original nodes in the first type of newly established nodes migrate the data records in the first part of sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of sub-partitions to the newly added nodes in the second type of newly established nodes, thereby migrating the data records in the current partition to the first new partition and the second new partition in units of sub-partitions, thereby realizing automatic expansion of the partition.

[0075] It can be seen that this manual introduces the concept of sub-partitions, which can logically divide the data records in the partition into sub-partitions. When the partition is subsequently expanded, the data records are migrated using sub-partitions as migration units, thereby achieving more flexible and convenient partition expansion, thereby better adapting to the growth of data volume and changes in business needs.

[0076] Figure 3 The present invention is a flowchart of another database partition expansion method provided by an exemplary embodiment.

[0077] In this embodiment, the process of partition expansion of database partition A is described in detail. Figure 3 , the partition expansion method may include the following steps:

[0078] Step 302: Obtain the total number of data records in database partition A.

[0079] Step 304 , determining whether the total number is greater than a partition capacity threshold, if so, executing step 306 .

[0080] In this embodiment, the total number of data records in database partition A may be obtained through a stored procedure trigger, and it may be determined whether the total number is greater than a partition capacity threshold.

[0081] Assume that the total number of data records in database partition A is 600, and the partition capacity threshold of database partition A is 400. After determining that the total number of data records in database partition A is greater than the partition capacity threshold, the expansion of database partition A can be triggered, and step 306 is executed.

[0082] If it is assumed that the partition capacity threshold of database partition A is 800, and the total number of data records of database partition A, 600, is less than the partition capacity threshold, there is no need to trigger the expansion of database partition A, and related operations for database partition A can continue to be performed, such as creating new data records.

[0083] Step 306, obtain several newly added nodes to obtain a node set.

[0084] Based on the aforementioned step 304, when it is determined that the total number of data records in the database partition A is greater than the partition capacity threshold, partition expansion is triggered, and a new node for partition expansion may be first obtained.

[0085] In this embodiment, it is assumed that the number of original nodes of database partition A is 4, that is, there are 4 original nodes jointly maintaining database partition A. ， Then, 4 new nodes can be obtained, and a node set including 8 nodes can be obtained.

[0086]

[0087] Table 2

[0088] Referring to the example in Table 2, when the database partition A is partitioned and expanded, the obtained node set includes four original nodes, namely, original node 1 to original node 4, and four newly added nodes, namely, newly added node 1 to newly added node 4.

[0089] Step 308: Use a distributed random algorithm to divide the nodes in the node set into a first type of newly established nodes and a second type of newly established nodes.

[0090] In this embodiment, on the one hand, new partitions can be created: database partition B1 and database partition B2. On the other hand, a distributed random algorithm can be used to divide the nodes in the node set shown in Table 2 into a first type of newly established nodes for maintaining database partition B1 and a second type of newly established nodes for maintaining database partition B2.

[0091] For example, see Figure 4 , through a distributed random algorithm, the original node 1, the original node 2, the newly added node 1 and the newly added node 2 are divided into the first type of newly established nodes for maintaining the database partition B1, and the original node 3, the original node 4, the newly added node 3 and the newly added node 4 are divided into the second type of newly established nodes for maintaining the database partition B2.

[0092] This embodiment uses a distributed random algorithm to perform node partitioning, which can ensure that the Byzantine nodes in the node set are allocated to newly established nodes of different categories, thereby ensuring the security of the database.

[0093] For example, assuming that the original node 1 and the newly added node 4 are Byzantine nodes, the distributed random algorithm can ensure that the original node 1 and the newly added node 4 are divided into different categories of newly established nodes, that is, the original node 1 and the newly added node 4 will not be simultaneously classified as the first category of newly established nodes, nor will they be simultaneously classified as the second category of newly established nodes. In this way, there is still at most one Byzantine node in the newly established nodes of each new partition. Based on the PBFT consensus protocol, each new partition can still reach consistency to ensure security.

[0094] Step 310, determining a sub-partition migration plan and triggering a consensus on the sub-partition migration plan.

[0095] Based on the aforementioned step 308, after the nodes in the node set are divided into database partition B1 or database partition B2, a sub-partition migration plan can be determined first, that is, migrating the first part of the sub-partitions in the current partition to database partition B1 and migrating the remaining second part of the sub-partitions in the current partition to database partition B2.

[0096] In this embodiment, it is assumed that database partition A includes six subpartitions, namely subpartitions 0001-0006. To achieve a better partition expansion effect, the subpartition migration plan can be determined to migrate subpartitions 0001-0003 to database partition B1, and migrate subpartitions 0004-0006 to database partition B2. In other words, after the migration, subpartitions 0001-0003 are maintained by database partition B1, and subpartitions 0004-0006 are maintained by database partition B2.

[0097] Next, a consensus on the migration plan may be triggered, that is, a consensus on migrating subpartitions 0001-0003 to database partition B1 and migrating subpartitions 0004-0006 to database partition B2 may be triggered.

[0098] Step 312: after consensus is reached, trigger the migration of data records of database partition B1 and database partition B2.

[0099] Based on the aforementioned step 310, the migration of data records can be performed after the entire network reaches consensus on migrating sub-partitions 0001-0003 to database partition B1 and migrating sub-partitions 0004-0006 to database partition B2.

[0100] In this embodiment, when performing migration, since each original node includes the data records in the six sub-partitions, for database partition B1, the original nodes in the first type of newly established nodes can migrate the data records in the first part of the sub-partitions to the newly added nodes in the first type of newly established nodes, that is, the original nodes migrate the data records in sub-partitions 0001-0003 to each newly added node in the first type of newly established nodes. For example, the original node 1 migrates the data records in sub-partitions 0001-0003 to the newly added node 1, and the original node 2 migrates the data records in sub-partitions 0001-0003 to the newly added node 2. In addition, the original node 1 and the original node 2 can also delete the data records in the second part of the sub-partitions, that is, the original node 1 and the original node 2 both delete the data records in sub-partitions 0004-0006, so that each node in database partition B1 only maintains the data records in sub-partitions 0001-0003.

[0101] Similarly, for database partition B2, the original node in the second type of newly established nodes can migrate the data records in the second part of the sub-partitions to the newly added nodes in the second type of newly established nodes, that is, the original node migrates the data records in sub-partitions 0004-0006 to each newly added node in the second type of newly established nodes. For example, the original node 3 migrates the data records in sub-partitions 0004-0006 to the newly added node 3, and the original node 4 migrates the data records in sub-partitions 0004-0006 to the newly added node 4. In addition, the original node 3 and the original node 4 can also delete the data records in the first part of the sub-partitions, that is, the original node 3 and the original node 4 both delete the data records in sub-partitions 0001-0003, so that each node in database partition B2 only maintains the data records in sub-partitions 0004-0006.

[0102] At this point, the migration of data records in the new partition database partition B1 and database partition B2 can be completed.

[0103] It should be noted that in other examples, the original node may not delete the data records in the sub-partition maintained in another new partition. For example, the original node 1 does not delete the data records in the sub-partitions 0004-0006, etc. This specification does not impose any special restrictions on this.

[0104] Step 314: for each sub-partition, according to the data record migration result, the partition corresponding to each sub-partition in the mapping relationship is updated from database partition A to database partition B1 or database partition B2.

[0105] In this embodiment, after the data record migration is completed, the mapping relationship between the sub-partitions and the partitions may be updated according to the migration result to guide the routing.

[0106]

[0107] Table 3

[0108] Before the partition expansion of database partition A, subpartitions 0001-0006 correspond to database partition A. After the partition expansion, please refer to Table 3. Subpartitions 0001-0003 correspond to database partition B1, and subpartitions 0004-0006 correspond to database partition B2.

[0109] It should be noted that, in this embodiment, when partition expansion is triggered, the transactions that have not been completed for the partition (for example, transactions for newly added data records) can be cached, and after the mapping relationship is updated, the transactions can be continued to be executed according to the mapping relationship. Assuming that after the partition expansion, the data records in sub-partition 0004 are migrated to database partition B2, the cached related transactions can be routed to the node where database partition B2 is located based on the mapping relationship.

[0110] It can be seen that after the data records are migrated, this manual can also update the mapping relationship between sub-partitions and partitions according to the migration results, thereby guiding the routing of database transactions or messages without affecting the normal service of the database.

[0111] This specification also provides a database access solution, which can implement database transaction or message routing based on the mapping relationship between sub-partitions and partitions.

[0112] Figure 5 It is a flowchart of a database access method provided by an exemplary embodiment.

[0113] The database access method provided in this embodiment can be applied in a routing component, and the routing component includes any object in the database system that has functions such as routing access requests, transactions, messages, etc. For example, the routing component can be a database agent, the routing component can also be a database node, etc., and this specification does not impose any special restrictions on this.

[0114] Please refer to Figure 5 , the database access method may include the following steps:

[0115] Step 502: Receive a database access request, wherein the database access request carries information of a target data record.

[0116] In this embodiment, the database access request may be a SQL (Structured Query Language Request) request, which carries information about the data record to be accessed (hereinafter referred to as the target record). Taking the modification of the account balance as an example, the access request may carry the account name of the target account.

[0117] Step 504: query the target sub-partition to which the target data record belongs according to the information of the target data record.

[0118] In this embodiment, taking LSM Tree (Log-Structured Merge Tree) as the database storage structure as an example, the sub-partition information to which each data record belongs can be added to the primary key of the corresponding data record in the SSTable. Taking the primary key as 32 bytes as an example, the first two bytes can be used to record the sub-partition information (such as the sub-partition identifier), and the following 30 bytes are used to identify the only first data record, for example, a unique user account.

[0119] In this embodiment, the unique identifier of the target data record can be determined based on the information of the target data record carried in the access request, and then the primary key of the target data record can be found based on the unique identifier, and the sub-partition to which the target record belongs can be obtained from the first two bytes of the primary key, which is called the target sub-partition.

[0120] Of course, in some examples, the unique identifier of the target record may also be directly carried in the access request as the information of the target record, and this specification does not impose any special limitation on this.

[0121] Step 506: query the target partition corresponding to the target subpartition according to the mapping relationship between the subpartitions and the partitions.

[0122] Based on the aforementioned step 504, after the target sub-partition to which the target data record belongs is obtained, the target partition corresponding to the target sub-partition may be searched according to the mapping relationship between sub-partitions and partitions maintained by the partition management component.

[0123] Please refer to the example in the above Table 3. Assuming that the sub-partition to which the target data record belongs is 0001, the corresponding target partition is found to be database partition B1.

[0124] Step 508: Send the database access request to the database node that maintains the target partition.

[0125] Based on the search result of the aforementioned step 506, the database access request may be sent to the database node maintaining the target partition, for example, the access request may be sent to the original node 3 maintaining the database partition B1.

[0126] Among them, after determining the target partition, the access request can be sent to a database node among multiple database nodes maintaining the target partition based on the type of the access request or load balancing strategies. The processing and implementation of this part can refer to related technologies.

[0127] As can be seen from the above description, this specification introduces the concept of sub-partitions. When performing database access routing, the target sub-partition to which the target data record belongs can be queried based on the information of the target data record, and then the target partition corresponding to the target sub-partition can be queried based on the mapping relationship between the sub-partition and the partition, and the database access request can be sent to the database node that maintains the target partition, thereby realizing intelligent routing of database access requests.

[0128] Figure 6 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0129] Please refer to Figure 7 The database partition expansion device 700 can be applied to Figure 6 In the device shown in the figure, the technical solution of this specification is implemented. Wherein, each data record in the partition belongs to a sub-partition, and the database partition expansion device 700 can be specifically applied to the partition management component, including:

[0130] A quantity acquisition unit 702 is used to acquire the total quantity of data records in a current partition, where the current partition includes multiple sub-partitions.

[0131] The set determining unit 704 acquires a number of newly added nodes to obtain a node set when the total number is greater than the partition capacity threshold, wherein the node set includes the number of newly added nodes and a number of original nodes maintaining the current partition;

[0132] A node division unit 706 divides the nodes in the node set into a first type of newly established nodes and a second type of newly established nodes, wherein the first type of newly established nodes maintains a first new partition, and the second type of newly established nodes maintains a second new partition, and both the first type of newly established nodes and the second type of newly established nodes include both the original nodes and the newly added nodes;

[0133] A migration determination unit 708 is configured to determine a subpartition migration plan for the multiple subpartitions in the current partition, wherein the subpartition migration plan includes migrating a first portion of the multiple subpartitions to the first new partition and migrating a second portion of the multiple subpartitions to the second new partition.

[0134] The record migration unit 710 triggers the migration of new partition data records so that the original nodes in the first type of newly established nodes migrate the data records in the first part of the sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of the sub-partitions to the newly added nodes in the second type of newly established nodes.

[0135] Optionally, the partition management component is further used to maintain a mapping relationship between sub-partitions and partitions, and the device further includes:

[0136] The mapping updating unit 712 updates, for each sub-partition, the partition corresponding to the sub-partition in the mapping relationship from the current partition to the first new partition or the second new partition according to the data record migration result.

[0137] Optionally, the process in which the node division unit 706 divides the nodes in the node set into first-category newly established nodes and second-category newly established nodes includes: using a distributed random algorithm to divide the nodes in the node set into first-category newly established nodes and second-category newly established nodes.

[0138] Optionally, the migration determination unit 708: after determining the sub-partition migration plan, triggers a consensus on the sub-partition migration plan; after the sub-partition migration plan is reached by consensus, the record migration unit 710 executes a process of triggering the migration of new partition data records.

[0139] Optionally, the record migration unit 710, during the process of migrating data records to a new partition, each original node in the first type of newly established nodes deletes the data records in the second part of the sub-partitions, and each original node in the second type of newly established nodes deletes the data records in the first part of the sub-partitions.

[0140] Figure 8 It is a block diagram of a database access device provided by an exemplary embodiment.

[0141] Please refer to Figure 8 This specification also provides a database access device 800, which can be applied to Figure 6 The device shown in the figure is used to implement the technical solution of this specification. The database access device 800 may include:

[0142] A request receiving unit 802 receives a database access request, wherein the database access request carries information of a target data record;

[0143] A sub-partition query unit 804 queries a target sub-partition to which the target data record belongs according to information of the target data record;

[0144] A partition query unit 806 queries a target partition corresponding to the target subpartition according to a mapping relationship between subpartitions and partitions;

[0145] A request forwarding unit 808 sends the database access request to a database node maintaining the target partition;

[0146] Optionally, the subpartition record to which each data record belongs is recorded in the primary key of the data record, and the process in which the subpartition query unit 804 queries the target subpartition to which the target data record belongs according to the information of the target data record includes:

[0147] Determine a unique identifier of the target data record according to the information of the target data record;

[0148] The primary key of the target data record is searched according to the unique identifier, and the target sub-partition to which the target data record belongs is obtained from the primary key.

[0149] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0150] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0151] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A method for expanding the capacity of a database partition, wherein each data record in the partition belongs to a sub-partition, and the method is applied to a partition management component, comprising: Obtain the total number of data records in the current partition, where the current partition includes multiple sub-partitions; When the total number is greater than the partition capacity threshold, a number of newly added nodes are acquired to obtain a node set, where the node set includes the number of newly added nodes and a number of original nodes that maintain the current partition; Dividing the nodes in the node set into a first type of newly established nodes and a second type of newly established nodes, wherein the first type of newly established nodes maintains a first new partition, and the second type of newly established nodes maintains a second new partition, and both the first type of newly established nodes and the second type of newly established nodes include both the original nodes and the newly added nodes; Determine a subpartition migration plan for the multiple subpartitions in the current partition, the subpartition migration plan comprising migrating a first portion of the multiple subpartitions to the first new partition and migrating a second portion of the multiple subpartitions to the second new partition; Trigger the migration of new partition data records so that the original nodes in the first type of newly established nodes migrate the data records in the first part of the sub-partitions to the newly added nodes in the first type of newly established nodes, and the original nodes in the second type of newly established nodes migrate the data records in the second part of the sub-partitions to the newly added nodes in the second type of newly established nodes.

2. The method according to claim 1, wherein the partition management component is further used to maintain a mapping relationship between sub-partitions and partitions, and the method further comprises: For each sub-partition, according to the data record migration result, the partition corresponding to the sub-partition in the mapping relationship is updated from the current partition to the first new partition or the second new partition.

3. The method according to claim 1, wherein the process of dividing the nodes in the node set into the first type of newly established nodes and the second type of newly established nodes comprises: The nodes in the node set are divided into a first type of newly established nodes and a second type of newly established nodes by using a distributed random algorithm.

4. The method according to claim 1, after determining the sub-partition migration plan, the method further comprises: Triggering consensus on sub-partition migration plans; After consensus is reached on the sub-partition migration plan, a process of triggering the migration of new partition data records is executed.

5. According to the method described in claim 1, during the migration of data records in the new partition, each original node in the first type of newly established nodes deletes the data records in the second part of the sub-partition, and each original node in the second type of newly established nodes deletes the data records in the first part of the sub-partition.

6. A database access method, applied to a routing component, comprising: receiving a database access request, wherein the database access request carries information of a target data record; Query the target sub-partition to which the target data record belongs according to the information of the target data record; According to the mapping relationship between sub-partitions and partitions, query the target partition corresponding to the target sub-partition; Sending the database access request to a database node maintaining the target partition; The target partition is expanded by using the method described in any one of claims 1 to 5.

7. The method according to claim 6, wherein the sub-partition to which each data record belongs is recorded in the primary key of the data record, and the process of querying the target sub-partition to which the target data record belongs according to the information of the target data record comprises: Determine a unique identifier of the target data record according to the information of the target data record; The primary key of the target data record is searched according to the unique identifier, and the target sub-partition to which the target data record belongs is obtained from the primary key.

8. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images