Database splitting method supporting dynamic capacity expansion and contraction

Abstract

The application relates to the technical field of databases, and provides a database splitting method supporting dynamic capacity expansion and reduction. The main purpose is to solve the data migration problem, improve the data migration efficiency, and solve the data positioning problem. The main scheme comprises the following steps: a compressed index tree with a special structure is designed, which is used for representing the association relationship among multiple database nodes; an addressing path table is added, which is used for recording the data historical storage strategy; a dynamic capacity expansion and reduction database quantity splitting method is designed, so that dynamic adjustment is realized in the database capacity expansion and reduction process; and a method for correctly accessing data is realized, so that the data query efficiency is improved. Through the above technical scheme, the database can be dynamically expanded and reduced, and the flexibility and adaptability of the database are improved.

Description

Technical Field

[0001] This invention relates to the field of database technology and provides a database sharding method that supports dynamic scaling. Background Technology

[0002] In application software systems, when a single database cannot meet the requirements for storage capacity, access performance, etc., a common practice is to shard the database. The principle is to prepare multiple databases, each acting as a shard node, and then store data that conforms to the same rules into the same shard node according to a specific algorithm.

[0003] However, simple database sharding still cannot perfectly solve the performance bottleneck problem caused by databases. If too many databases are designed from the beginning, resources will be wasted. Conversely, if the number of databases is set too small at the beginning, performance requirements will still not be met as the data volume increases. Therefore, database sharding needs to support dynamic expansion and contraction.

[0004] One approach involves hashing the specified sharding field content of the data object to be stored, then taking the modulo of the hash result with the number of database sharding nodes. The modulo result is the target database sharding number, and the data is then stored in that sharding node. When database performance reaches a bottleneck, a new sharding node is added, and all the data is retrieved and re-stored using the same algorithm. With more sharding nodes, more storage space is available, and more databases can handle concurrent requests, thus reducing the read / write pressure on individual databases and achieving database expansion.

[0005] One approach is to randomly assign integer numbers to all database shards. These numbers cannot be repeated and can be non-consecutive. Then, the content of the specified shard field of the data object to be stored is hashed. The hash result is compared sequentially with the integer values ​​of all database shard numbers. The data is then stored in the database shard with the number largest but closest to the hash value. When database performance reaches a bottleneck, a new database shard is added. This new shard is also randomly assigned an integer number, which cannot be duplicated. With the storage strategy unchanged, data that would have been stored in a shard with a larger number but is smaller than the new shard number will be stored in the new shard. This distributes the storage load across the original shard, thus achieving database expansion.

[0006] While existing methods can theoretically expand database capacity, reduce the pressure on individual databases, and alleviate the problem of degraded write performance caused by the continuous growth of data volume and concurrent access, they still have some problems in practical application scenarios, mainly in the following aspects:

[0007] (1) Data migration problem: During the expansion and contraction of the database, data needs to be migrated from one shard node to another. This process requires recalculation and restoring of the original data, which may cause some data queries to fail. It may also have a significant impact on the operation of the application software. The entire data migration process takes a long time to execute, which may cause the application software to be unable to provide normal services, or even require interruption of software services to complete the data migration.

[0008] (2) Data location problem: Before and after database expansion and contraction, the storage locations of data are not linked. After database expansion, it is impossible to track the storage location of certain data before the expansion, and it is impossible to roll back the database expansion operation. If the expansion process cannot be rolled back, it will bring significant risks. If the expansion fails or data is lost due to data transfer during the expansion process, it will be impossible to restore the data to its original location. Secondly, it also makes database contraction operations more difficult.

[0009] (3) Data association issues before and after expansion and shrinkage: During the expansion and shrinkage process, no association relationship is established between the data in the old and new databases. If it is necessary to revert to the state before expansion after the expansion and shrinkage is completed, there will be a major data location problem. Summary of the Invention

[0010] The main technical problems to be solved by this invention include: how to solve the data migration problem and improve the data migration efficiency; how to solve the data location problem and improve the data location efficiency; and how to establish the correlation between data before and after expansion and contraction to facilitate rollback operations.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical means:

[0012] A database sharding method that supports dynamic scaling includes the following steps:

[0013] Step 1: Construct a compressed index tree to represent the relationship between multiple database nodes. In the compressed index tree, each node corresponds to a database shard. During the expansion and contraction of the database, the tree structure is used to represent the splitting of child nodes by the parent node and the merging of child nodes into the parent node. The tree structure expresses the splitting relationship between nodes.

[0014] Step 2: Use an addressing path table to record the historical data storage strategy. Each sub-database node can quickly locate the accurate location of the existing data before expansion or reduction by using the historical addressing path in the addressing path table, even if the addressing path changes dynamically before and after the expansion or reduction process.

[0015] Step 3: Using a compressed index tree and addressing path table, a method for dynamically expanding and shrinking the number of database shards is implemented, specifically as follows:

[0016] When a database shard needs to be expanded, a new shard is added by splitting the parent node into child nodes. At the same time, the index tree and address path table are updated, and the new shard is associated with the original node to achieve expansion. When it needs to be reduced, it is achieved by reducing the number of shards. The index tree and address path table are updated, and the existing data in the node to be deleted is migrated to the merge node to complete the reduction.

[0017] Step 4: During the dynamic scaling up and scaling down process, the asynchronous data migration method is used to enable data access without stopping the service.

[0018] In the above calculation scheme, the compressed index tree includes:

[0019] Database shard number: identifies the corresponding database shard, such as Figure 1 a, c, e, g;

[0020] The hash prefix of this node: the addressing path from the root node to each node. If the hash value matches the hash prefix of the current node, then the database shard corresponding to this node is used.

[0021] Right child node pointer list: An array of pointers stores pointers to right child nodes at different levels;

[0022] Addressing path: The hash prefix from the root node to the node is the addressing path of that node;

[0023] Effective timestamp: The time point at which each expansion or contraction operation affects the change of the node addressing path, i.e., the effective timestamp of the addressing path;

[0024] Addressing Path Table: A historical record of all addressing paths on each node and their corresponding effective timestamps.

[0025] In the above technical solution, the dynamic expansion process of the database consists of the following steps:

[0026] a1. Identify the current node as the database shard that needs to be expanded;

[0027] a2. Prepare a new database shard node as the new node, and establish the association between the current node and the new node, that is, the pointer of the current node points to the new node;

[0028] a3. Set the addressing path of the new node. The addressing path of the new node is the original addressing path of the current node plus 1 to the right.

[0029] a4. Update the addressing path of the current node. The new addressing path of the current node is obtained by adding 0 to the right of the original addressing path.

[0030] a5. Refresh the addressing path table, adding the current node's addressing path and current timestamp, and the new node's addressing path and current timestamp to the addressing path table.

[0031] In the above technical solution, the dynamic scaling down process of the database consists of the following steps:

[0032] b1. Determine the database shard nodes that need to be scaled down as the nodes to be deleted. The pointer list of the nodes to be deleted must be empty, that is, the nodes to be deleted do not point to any nodes.

[0033] b2. Set the node containing the pointer to the node to be deleted as the merge node in the shrinking process;

[0034] b3. Set the addressing path of the merged node. Remove the rightmost 0 from the current addressing path of the merged node to get the new addressing path of the merged node.

[0035] b4. Refresh the addressing path table, adding the new addressing path of the merged node and the current timestamp to the addressing path table;

[0036] b5. Asynchronous data migration: migrate existing data in the node to be deleted to the merge node during any idle time of the system.

[0037] b6. Delete the pointer of the merged node to the node to be deleted, and remove the node to be deleted from the entire system.

[0038] In the above technical solution, during the dynamic expansion and contraction process, correct data access is achieved through a target data addressing method, which consists of the following steps:

[0039] Step c1. The hash prefix is ​​completely matched to the addressing path, and the current node is located according to the search rules;

[0040] Step c2. Verify the effective timestamp, compare the data entry time with the effective time of the matching address path in the address path table.

[0041] If the data time is greater than the currently matched effective timestamp, then the current node is the data storage node, and the address search ends;

[0042] If the data time is not greater than the currently matched effective timestamp, it is necessary to further determine whether the current node has a list of pointers pointing to the next node.

[0043] If it exists, continue matching the addressing path of the next node pointed to by the pointer, and continue to execute steps c1-c2;

[0044] If it does not exist, backtrack to the previous node, match the previous addressing path in its addressing path table, and continue to execute steps c1-c2; until the data time in step c2 is greater than the effective timestamp of the currently matched addressing path, then the current node is the data storage node, and the addressing ends.

[0045] In the above technical solution, the asynchronous data migration method during the database scaling-down process,

[0046] Before migrating data during the downsizing process, it is necessary to first update the addressing paths of the merged nodes and the addressing path table;

[0047] According to the target data addressing method, no new data will be inserted into the node to be deleted;

[0048] Before all the data of the nodes to be deleted is migrated, this stage will only perform query, deletion and update operations on the existing data of the nodes to be deleted;

[0049] During this phase, there is no need to stop the service. At any time during off-peak hours, all existing data of the nodes to be deleted will be migrated to the merged nodes.

[0050] Once all existing data for the node to be deleted has been migrated, all services for the node to be deleted can be stopped and the node to be deleted can be removed from the index tree.

[0051] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:

[0052] First, it solves the data migration problem: By designing a compressed index tree with a special structure and adding an addressing path table, it is possible to achieve data migration without recalculation and re-storage during database expansion and contraction, thereby reducing the time and cost of data migration and improving the efficiency of data migration.

[0053] Second, it solves the data location problem: by using an addressing path table to record the historical data storage strategy, even if the addressing path changes dynamically before and after expansion or reduction, the data storage node can be quickly located based on the historical addressing path, thus improving the efficiency of data location.

[0054] Third, dynamic expansion and contraction are achieved: By designing a method for dynamically expanding and contracting the number of database shards, the number of database shards can be dynamically adjusted according to the real-time needs of the database, realizing the dynamic expansion and contraction of the database, and improving the flexibility and adaptability of the database.

[0055] Fourth, improved data query efficiency: By designing the correct data access method, data queries are still performed through the original addressing strategy during dynamic expansion and contraction, without affecting query efficiency, thus improving the speed and accuracy of data queries.

[0056] Fifth, asynchronous data migration was implemented: During the downsizing process, the existing data in the node to be deleted was migrated to the merged node through an asynchronous data migration method, which reduced the impact of data migration on the application system and improved the stability of the downsizing process.

[0057] In summary, this invention solves problems such as data migration, data location, dynamic expansion and contraction, and data query by employing a specially structured compressed index tree, addressing path table, and dynamic expansion and contraction database number methods. It realizes the dynamic expansion and contraction of the database, improves the flexibility and adaptability of the database, and provides an efficient, stable, and reliable data storage and management solution for application systems. Attached Figure Description

[0058] Figure 1 It is an index tree with a complete binary tree structure;

[0059] Figure 2 This is a schematic diagram of a compressed index tree structure;

[0060] Figure 3 Workflow diagram for database expansion;

[0061] Figure 4 Database scaling down workflow diagram;

[0062] Figure 5 Workflow for addressing target data. Detailed Implementation

[0063] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0064] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0065] While these existing methods can theoretically expand the database, reduce the pressure on a single database, and alleviate the problem of declining write performance caused by the continuous growth of data volume and concurrent access, they all have the following defects and shortcomings in practical application scenarios.

[0066] 1. Both database expansion and contraction require migrating the original data, which can lead to some data query failures. This involves migrating some or even all of the data from the original database node and storing it in the new database node according to the new calculation results. This process has a significant impact on the operation of the software application. The entire data migration process takes a long time to complete. During this process, the application software cannot provide normal services and may even need to be interrupted to complete the data migration. Services can only be resumed after the entire migration process is completed.

[0067] 2. Before and after database expansion or shrinking, no correlation is established between the storage locations of the data. Once the database expands, it becomes impossible to track the storage location of data before the expansion, making it impossible to roll back the expansion operation. If the expansion process cannot be rolled back, it poses a significant risk. If expansion fails or data is lost due to data transfer during the expansion process, it is impossible to restore the data to its original location. Furthermore, it significantly complicates database shrinking operations.

[0068] In view of the above-mentioned shortcomings, the present invention proposes the following two objectives:

[0069] By employing a specialized data design, this system overcomes the data migration and service interruption issues that arise when adding a new database requires recalculating the storage location of data already stored in the original database. It records the precise time data was added to the database by timestamping data objects. After adding a new database, its startup time is also recorded. By comparing the data's addition time with the database expansion time, it can be determined whether a particular piece of data was added before or after the database was added. This ensures that even when a database is added, there is no need to migrate the original data; the database node containing a specific piece of data can be retrieved immediately.

[0070] By introducing a specially optimized index structure, the problem of data inconsistency and lack of traceability before and after expansion is overcome when a new database is added without establishing a relationship between the new database and the original database requiring expansion. This leads to problems such as the inability to roll back expansion operations, difficulty in scaling down, and difficulty in locating and tracing data. When a database is under excessive pressure and needs to be expanded to alleviate it, after adding a new database node, the new database node and the original database node will establish a direct relationship through an algorithm. Once the new database is started, it begins to share some of the data that should have been stored in the associated original database, thereby reducing the data growth rate of the original shard node and distributing the pressure of concurrent access, achieving the purpose of expansion. At the same time, it is possible to trace the original database node where the data in the new database should have been stored before expansion. Therefore, during scaling down, this part of the data can be quickly transferred to the original database node, realizing the dynamic expansion and shrinking of the database.

[0071] This invention overcomes the data migration and service interruption problems caused by the need to recalculate the storage location of data already stored in the original database when adding a new database, through a special data design. By timestamping data objects, the exact time of data entry is recorded. After adding a new database, the startup time of the new database is also recorded. By comparing the data entry time with the database expansion time, it can be determined whether a certain data was entered before or after the addition of the new database. This ensures that even if a database is added, the original data can be retrieved without migration, and the database node where the data resides can be obtained immediately.

[0072] By introducing a compressed index tree structure, the problem of data inconsistency and untraceability before and after expansion is overcome when a new database is added without establishing a relationship between the new database and the original database requiring expansion. This leads to problems such as the inability to roll back expansion operations and difficulty in scaling down, as well as difficulty in locating and tracing data. When a database is under excessive pressure and needs to be expanded to alleviate it, after adding a new database node, the new database node and the original database node will establish a direct relationship through an algorithm. Once the new database is started, it begins to share some of the data that should have been stored in the associated original database, thereby reducing the data growth rate of the original shard node and distributing the pressure of concurrent access, achieving the purpose of expansion. At the same time, it is possible to trace the original database node where the data in the new database should have been stored before expansion. Therefore, during scaling down, this part of the data can be quickly transferred to the original database node, realizing the dynamic expansion and shrinking of the database.

[0073] To facilitate a better understanding of the technical solution of this invention by those skilled in the art, the following specific description is provided:

[0074] I. Overall Structure

[0075] The core of this invention includes the following concepts:

[0076] Index tree: A data structure that stores the mapping relationship between hash key values ​​and corresponding database shards;

[0077] Data access: The process of finding the database shard node in the index tree based on the hash key value and accessing the data;

[0078] Addressing path: The path identifier from the root node to each node in the index tree. A single path from the parent node to the child node is represented by 0 on the left and 1 on the right. Each time a node is added, the path is incremented by one bit to the right.

[0079] Timestamp: Records the time point in the index tree when the addressing path of related nodes changes during the splitting or merging process;

[0080] Addressing Path Table: A record table of addressing path updates during each split or merge process of each node. Each record contains an addressing path and the effective time of that addressing path.

[0081] Dynamic expansion: When the capacity of a certain database shard becomes too large, the process of adding database shard nodes to expand the capacity;

[0082] Dynamic scaling down: When the capacity of a certain database shard is too small, the database shard nodes are merged to perform scaling down;

[0083] II. Index Tree Structure

[0084] An index tree is a data structure that stores the mapping relationship between hash key values ​​and corresponding database shards.

[0085] Essentially, the index tree proposed in this invention is a compressed form of a complete binary tree. To more conveniently describe the index tree, we will first describe the tree structure in its uncompressed complete binary tree form.

[0086] Figure 1 This is an index tree with a complete binary tree structure. In this complete binary tree, each leaf node represents a database partition (here, each node is identified by "a, b, c, ..."). The path from the root node to that node (i.e., the binary 01 string after the colon) is the prefix of the matching hash key value. The left and right child nodes of each node in this complete binary tree are represented by 0 on the left and 1 on the right. Figure 1 In essence, it describes four sub-databases: d, f, g, and c.

[0087] When looking up the corresponding database shard using a hash key, follow these steps:

[0088] 1. Set the current node as the root node.

[0089] 2. If the current node is a leaf node, the search ends, and the database shard corresponding to the current node is the result.

[0090] 3. Take each bit of the hash key value in sequence.

[0091] 4. If this bit is 0, then set the current node as its left child; otherwise, set it as its right child.

[0092] 5. Repeat steps 2-4 until a leaf node is reached.

[0093] If a database needs to be expanded, create the left and right child nodes of the corresponding leaf node, and match one more bit when searching by hash key.

[0094] The reduction process is similar.

[0095] As can be seen, in the index tree, only leaf nodes correspond to database shards, while non-leaf nodes are only used in the search process. This leads to two drawbacks:

[0096] 1. When the index tree is large, a significant amount of space is wasted;

[0097] 2. During expansion and contraction, the node splitting process cannot be accurately recorded, i.e., which node each node split from. This splitting process information is crucial for smooth data migration.

[0098] Compressed index tree

[0099] During database expansion, for efficiency, two new database shards are not introduced to replace the original database shards. Instead, one new database shard is introduced to share some of the pressure on the original database shards. Therefore, in reality, only the leaf nodes of this tree correspond to the database shards.

[0100] The index tree structure proposed in this invention is a compression of the complete binary tree index structure. Each node corresponds to a shard. During expansion, a new node is introduced to share the workload of the original node, becoming its right child, while the original node becomes its left child. Relative relationships must be maintained to express the splitting process (relationships are needed for degradation operations during subsequent accesses). In this structure, a node's left child is merged into the node itself. Figure 1 In the compressed structure corresponding to the complete binary tree structure, nodes b and d will be merged into node a, and node f will be merged into node e. Additionally, an array of pointers is needed to store pointers to the right children at different levels.

[0101] As can be seen, in the compressed index tree, each node, whether a leaf node or a non-leaf node, represents an actual database partition. At the same time, the path from the root node to the current node (i.e., the current addressing path of the node) also describes the process of node splitting.

[0102] Database shard number: identifies the corresponding database shard.

[0103] The hash prefix of this node: The addressing path from the root node to each node; if the hash value matches the hash prefix of the current node, then the database shard corresponding to this node is used.

[0104] List of right child node pointers: An array of pointers stores pointers to right children at different levels.

[0105] Addressing Path and Effective Timestamp: The hash prefix from the root node to each node is its addressing path. The time when each expansion or contraction operation affects the change of the node's addressing path is the effective timestamp of that addressing path.

[0106] Addressing Path Table: A historical record of all addressing paths on each node and their corresponding effective timestamps.

[0107] Example of an addressing path table:

[0108]

[0109] Search process

[0110] Step 1. Set the current node as the root node.

[0111] Step 2. Compare starting from the 0th bit of the hash key value.

[0112] Step 3. If the hash key value exactly matches the hash prefix of the current node, the search ends, and the database shard corresponding to the current node is the result.

[0113] Step 4. If the current bit of the hash key is 0, continue comparing the next bit.

[0114] Step 5. Otherwise (the current bit of the hash key is 1), search for the corresponding level pointer in the right child pointer list, find the next node, and set it as the current node.

[0115] Step 6. Repeat steps 3-5 until a node that exactly matches the hash prefix is ​​found.

[0116] Step 7. Arrange the nodes visited during the search process in sequence to form a node sequence.

[0117] The node sequence described in step 7 precisely represents the node splitting process and can be used for rollback operations in the subsequent data access process.

[0118] III. Methods for Dynamic Database Expansion

[0119] When a database shard node experiences excessive load, it needs to be expanded. The expansion process involves adding new shard nodes, updating the index tree, and updating the addressing path tables of relevant nodes. The database expansion process workflow is as follows: Figure 3 As shown.

[0120] The database expansion process consists of the following steps:

[0121] Step 1. Identify the database shard node that needs to be expanded as the current node.

[0122] Step 2. Prepare a new database shard node as the new node, and establish the association between the current node and the new node, that is, the pointer of the current node points to the new node.

[0123] Step 3. Set the addressing path for the new node. The addressing path for the new node is the original addressing path of the current node plus 1 to the right.

[0124] Step 4. Update the addressing path of the current node. The new addressing path of the current node is obtained by adding 0 to the right of the original addressing path.

[0125] Step 5. Refresh the addressing path table, adding the current node's addressing path and current timestamp, and the new node's addressing path and current timestamp to the addressing path table.

[0126] III. Methods for Dynamically Shrinking Databases

[0127] When the overall pressure on the database system decreases (e.g., after the database concurrency pressure has passed its peak and gradually eased), it is necessary to scale down the database system. The entire scaling-down process involves reducing the number of sharded nodes, updating the index tree, and updating the addressing path tables of relevant nodes. The workflow of the database scaling-down process is as follows: Figure 4 As shown.

[0128] The database scaling down process consists of the following steps:

[0129] Step 1. Identify the database shard nodes that need to be scaled down as the nodes to be deleted. The pointer list of the nodes to be deleted must be empty, that is, the nodes to be deleted do not point to any other nodes.

[0130] Step 2. Set the node containing the pointer to the node to be deleted as the merge node in the scaling-down process.

[0131] Step 3. Set the addressing path of the merged node. Remove the rightmost 0 from the current addressing path of the merged node to get the new addressing path for that node.

[0132] Step 4. Refresh the addressing path table, adding the new addressing path of the merged node and the current timestamp to the addressing path table.

[0133] Step 5. Asynchronous data migration: During any idle time of the system (such as when the transaction volume is low in the early morning), migrate the existing data in the node to be deleted to the merge node.

[0134] Step 6. Delete the pointers from the merged node that point to the node to be deleted, and remove the node to be deleted from the entire system.

[0135] IV. Data Addressing Methods under Dynamic Expansion and Scaling

[0136] The target data addressing process consists of the following steps:

[0137] Step 1. The hash prefix is ​​a complete match for the addressing path. Based on the lookup rules, the current node is located.

[0138] Step 2. Verify the timestamp. Compare the data time with the effective timestamp of the matching addressing path in the addressing path table. If the data time is greater than the effective timestamp of the currently matched addressing path, then the current node is the data storage node, and the addressing ends.

[0139] Step 3. If the data time in step 2 is not greater than the effective timestamp of the currently matched addressing path, it is necessary to further determine whether the current node has a pointer list pointing to the next node. If it exists, continue to match the addressing path of the next node pointed to by the pointer and continue to execute steps 1-2; if it does not exist, backtrack to the previous node, match the previous addressing path in its addressing path table, and continue to execute steps 1-2; until the data time in step 2 is greater than the effective timestamp of the currently matched addressing path, then the current node is the data storage node, and the addressing ends.

[0140] V. Asynchronous Data Migration Methods During Downsizing

[0141] During database downsizing, the existing data in the nodes to be deleted is migrated to the merged nodes. Typically, data migration in a system affects the stability of data queries by application systems because if services are not stopped during the migration process, the application system cannot accurately know which shard the target data being queried is currently in, leading to query failures. In this invention, the data migration involved in the downsizing process is processed asynchronously; data queries still use the original addressing strategy, thus not affecting query issues during the migration process.

[0142] Before the data migration in the scaling-down phase, the addressing path and addressing path table of the merge node have been updated. According to the data addressing strategy, no new data will be inserted into the node to be deleted until all the data of the node to be deleted has been migrated. During this phase, only the existing data of the node to be deleted will be queried, deleted, and updated. There is no need to stop the service during this phase. At any off-peak time (such as when the transaction volume is low in the early morning), all the existing data of the node to be deleted will be migrated to the merge node. Once all the existing data of the node to be deleted has been migrated, all services of the node to be deleted can be stopped and the node to be deleted can be removed from the index tree, thus completing the entire scaling-down process.

[0143] In summary, the key points of this invention include:

[0144] A compressed index tree with a special structure was designed to represent the relationships between multiple database nodes. Database expansion and contraction are both represented by splitting child nodes from a parent node in the tree structure and merging child nodes back into the parent node. The tree structure expresses the splitting relationships between nodes, which are the splitting relationships for data storage, retrieval, and location. This effectively represents the data storage relationships before and after the expansion and contraction process, facilitating data location and traceability, and avoiding problems such as being unable to find the original location of data after expansion or being unable to contract.

[0145] An addressing path table has been added to record historical data storage strategies. Each shard node can quickly locate the exact position of existing data before expansion or contraction by using the historical addressing paths in the addressing path table, even if the addressing paths change dynamically before and after expansion or contraction. This reduces the overhead of having to rely on trial and error to locate data, improves query efficiency, and reduces the frequency of database operations. In addition, the expansion or contraction process only requires updating the addressing path table, without requiring downtime or service interruption.

[0146] Using a compressed index tree structure and address path table structure, a method for dynamically expanding and shrinking the number of database shards was designed, as well as a method for correctly accessing data during the dynamic expansion and shrinking process.

Claims

1. A database sharding method supporting dynamic scaling, characterized in that: Includes the following steps: Step 1: Construct a compressed index tree to represent the relationships between multiple database nodes. In the compressed index tree, each node corresponds to a database shard. The compressed index tree includes: Database shard number: identifies the corresponding database shard; The hash prefix of this node: the addressing path from the root node to each node. If the hash value matches the hash prefix of the current node, then the database shard corresponding to this node is used. Right child node pointer list: An array of pointers stores pointers to right child nodes at different levels; Addressing path: The hash prefix from the root node to the node is the addressing path of that node; Effective timestamp: The time point at which each expansion or contraction operation affects the change of the node addressing path, i.e., the effective timestamp of the addressing path; Addressing Path Table: A historical record of all addressing paths on each node and their corresponding effective timestamps; During the expansion and contraction of a database, the splitting of child nodes by a parent node and the merging of child nodes into the parent node are represented by a tree structure. The tree structure expresses the splitting relationship between nodes. Step 2: Use an addressing path table to record the historical data storage strategy. Each sub-database node can quickly locate the accurate location of the existing data before expansion or reduction by using the historical addressing path in the addressing path table, even if the addressing path changes dynamically before and after the expansion or reduction process. Step 3: Using a compressed index tree and addressing path table, a method for dynamically expanding and shrinking the number of database shards is implemented, specifically as follows: When a database shard node needs to be expanded, a new shard node is added by splitting the parent node into child nodes. At the same time, the index tree and addressing path table are updated to establish a relationship between the new shard node and the original node, thereby achieving expansion. When scaling down is required, it is achieved by reducing the number of shard nodes, updating the index tree and addressing path table, and migrating the existing data in the nodes to be deleted to the merged nodes to complete the scaling down. Step 4: During the dynamic scaling up and scaling down process, asynchronous data migration is used to enable data access without stopping the service; During dynamic expansion and contraction, correct data access is achieved through the target data addressing method, which consists of the following steps: Step c1. The hash prefix is ​​completely matched to the addressing path, and the current node is located according to the search rules; Step c2. Verify the effective timestamp, comparing the data entry time with the effective timestamp of the matching addressing path in the addressing path table. If the data time is greater than the currently matched effective timestamp, then the current node is the data storage node, and the address search ends; If the data time is not greater than the currently matched effective timestamp, it is necessary to further determine whether the current node has a list of pointers pointing to the next node. If it exists, continue matching the addressing path of the next node pointed to by the pointer, and continue to execute steps c1-c2; If it does not exist, backtrack to the previous node, match the previous addressing path in its addressing path table, and continue to execute steps c1-c2; until the data time in step c2 is greater than the effective timestamp of the currently matched addressing path, then the current node is the data storage node, and the addressing ends.

2. The database sharding method supporting dynamic scaling according to claim 1, characterized in that: The dynamic expansion process of a database consists of the following steps: a1. Identify the current node as the database shard that needs to be expanded; a2. Prepare a new database shard node as the new node, and establish the association between the current node and the new node, that is, the pointer of the current node points to the new node; a3. Set the addressing path of the new node. The addressing path of the new node is the original addressing path of the current node plus 1 to the right. a4. Update the addressing path of the current node. The new addressing path of the current node is obtained by adding 0 to the right of the original addressing path. a5. Refresh the addressing path table, adding the current node's addressing path and current timestamp, and the new node's addressing path and current timestamp to the addressing path table.

3. The database sharding method supporting dynamic scaling according to claim 1, characterized in that: The dynamic scaling-down process of a database consists of the following steps: b1. Determine the database shard nodes that need to be scaled down as the nodes to be deleted. The pointer list of the nodes to be deleted must be empty, that is, the nodes to be deleted do not point to any nodes. b2. Set the node containing the pointer to the node to be deleted as the merge node in the shrinking process; b3. Set the addressing path of the merged node. Remove the rightmost 0 from the current addressing path of the merged node to get the new addressing path of the merged node. b4. Refresh the addressing path table, adding the new addressing path of the merged node and the current timestamp to the addressing path table; b5. Asynchronous data migration: migrate existing data in the node to be deleted to the merge node during any idle time of the system. b6. Delete the pointer of the merged node to the node to be deleted, and remove the node to be deleted from the entire system.

4. The database sharding method supporting dynamic scaling according to claim 1, characterized in that: Asynchronous data migration methods during database scaling down. Before migrating data during the downsizing process, it is necessary to first update the addressing paths of the merged nodes and the addressing path table; According to the target data addressing method, no new data will be inserted into the node to be deleted; Before all the data of the nodes to be deleted is migrated, this stage will only perform query, deletion and update operations on the existing data of the nodes to be deleted; During this phase, there is no need to stop the service. At any time during off-peak hours, all existing data of the nodes to be deleted will be migrated to the merged nodes. Once all existing data for the node to be deleted has been migrated, all services for the node to be deleted can be stopped and the node to be deleted can be removed from the index tree.

Images