Method, device and electronic device for modifying sharding rules of distributed database

Abstract

The present invention provides a method, device and electronic device for modifying sharding rules of a distributed database, which can be applied to the field of databases. The method includes: in response to detecting a sharding rule modification instruction, determining the original storage node and the new storage node related to the modification of the sharding rule, and the first data table to be modified in the original storage node; obtaining the sharding rule modification parameter; in the case of determining that the sharding rule modification parameter is a modification between storage nodes, creating a second data table with the same structure as the first data table in the new storage node; inserting the data in the first data table into the second data table through an insert statement, so as to realize data migration between the original storage node and the new storage node; exchanging the storage paths and sharding rules of the first data table and the second data table in the system table, so as to realize the online sharding rule modification of the distributed database.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and more specifically, to a method, device and electronic device for modifying sharding rules of a distributed database. Background Art

[0002] In the construction phase of a distributed database, the data sharding rules are specified when creating a data table. Therefore, when storing data in a data table, the data is stored according to the sharding rules. In the application phase of a distributed database, when running a business, if the sharding key or sharding method of the data table is inappropriate, the execution performance of SQL statements written based on the Structured Query Language (SQL) will be reduced. In addition, the execution performance of SQL statements is generally improved by modifying the sharding rules.

[0003] However, in a distributed database, sharding rules can be specified when creating a data table, but cannot be modified after the data table is created. This is because the modification of sharding rules will affect the data already stored in the data table, such as changes in storage nodes. Therefore, related technologies generally do not support online modification of sharding rules, which causes the modification of sharding rules to suspend the business of the distributed database and affect the user experience. Summary of the invention

[0004] In view of this, the present invention provides a method, device and electronic device for modifying sharding rules of a distributed database.

[0005] One aspect of the present invention provides a method for modifying sharding rules of a distributed database, comprising: in response to detecting a sharding rule modification instruction, determining an original storage node and a new storage node related to the modification of the sharding rule, and a first data table to be modified in the original storage node; obtaining a sharding rule modification parameter, wherein the sharding rule modification parameter is used to indicate the manner in which the distributed database modifies the sharding rule online; when it is determined that the sharding rule modification parameter is a modification between storage nodes, creating a second data table with the same structure as the first data table in a new storage node; inserting the data in the first data table into the second data table through an insert statement, so as to realize data migration between the original storage node and the new storage node; and exchanging the storage paths and sharding rules of the first data table and the second data table in the system table to realize online sharding rule modification of the distributed database.

[0006] According to an embodiment of the present invention, an insert statement includes a query sub-statement and an insert sub-statement, and the first data table includes at least one data record; through the insert statement, the data in the first data table is inserted into the second data table, including: through a first thread, based on the query sub-statement, real-time querying each data record from the first data table; through a second thread parallel to the first thread, based on the insert sub-statement, synchronously inserting the data records queried from the first data table into the second data table.

[0007] According to an embodiment of the present invention, the method further includes: after determining the first data table to be modified in the original storage node, adding a write-disable lock for the first data table, so that when the sharding rules are modified, the write operation to the first data table is stopped.

[0008] According to an embodiment of the present invention, the method further includes: after inserting the data in the first data table into the second data table, increasing the lock level for the first data table, so that when the sharding rules are modified, the write operation and query operation on the first data table are stopped.

[0009] According to an embodiment of the present invention, the method further includes: before raising the lock level for the first data table, in response to detecting a user query instruction for the first data table, querying target data records from the first data table according to query parameters in the query instruction.

[0010] According to an embodiment of the present invention, the method further includes: after exchanging the storage paths of the first data table and the second data table in the system table, deleting the first data table in the original storage node; and releasing the lock on the first data table.

[0011] According to an embodiment of the present invention, exchanging the storage paths of the first data table and the second data table in a system table includes: in the system table, exchanging the physical file name and sharding rules of the first data table with the physical file name and sharding rules of the second data table.

[0012] According to an embodiment of the present invention, it also includes: when it is determined that the sharding rule modification parameters are modified interactively by the computing node and the storage node, the data in the first data table is stored in the computing node; the data in the first data table is deleted; in the system table, the original sharding rule of the first data table is modified to a new sharding rule; and according to the new sharding rule, the data in the first data table stored in the computing node is stored in the new storage node.

[0013] Another aspect of the present invention provides an apparatus for modifying sharding rules of a distributed database, including: a determination module, for determining, in response to detecting a sharding rule modification instruction, an original storage node and a new storage node related to the modification of the sharding rules, and a first data table to be modified in the original storage node; an acquisition module, for acquiring sharding rule modification parameters, wherein the sharding rule modification parameters are used to indicate the manner in which the distributed database modifies the sharding rules online; a table creation module, for creating a second data table with the same structure as the first data table in a new storage node when determining that the sharding rule modification parameters are modified between storage nodes; a data migration module, for inserting the data in the first data table into the second data table through an insert statement, so as to realize data migration between the original storage node and the new storage node; and a path modification module, for exchanging the storage paths and sharding rules of the first data table and the second data table in the system table, so as to realize online sharding rule modification of the distributed database.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0016] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when being executed.

[0017] The embodiments of the present invention support multiple online sharding rule modifications through sharding rule modification parameters, which facilitates the flexible implementation of online sharding rules. In addition, based on the modification method between storage nodes, it supports direct data migration between the original storage node and the new storage node at the bottom layer, without pulling the data to the upper computing node, which has higher modification efficiency and faster modification speed. Since there is no need to occupy the data pulled from the original storage node in the computing node cache, the embodiments of the present invention can also occupy less computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture to which the method and apparatus for modifying sharding rules of a distributed database of the present invention can be applied is shown.

[0020] Figure 2A flow chart of a method for modifying sharding rules in a distributed database according to an embodiment of the present invention is shown.

[0021] Figure 3A A schematic diagram of a method for modifying sharding rules in the case of modification between storage nodes according to an embodiment of the present invention is shown.

[0022] Figure 3B A schematic diagram of a sharding rule modification method in the case of interactive modification of computing nodes and storage nodes according to an embodiment of the present invention is shown.

[0023] Figure 4 A block diagram of an apparatus for modifying sharding rules of a distributed database according to an embodiment of the present invention is shown.

[0024] Figure 5 A block diagram of an electronic device applicable to a method for modifying sharding rules of a distributed database according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] In order to at least partially solve the technical problems that it is inconvenient to modify the sharding rules of a distributed database online and the modification occupies a large number of computing resources, an embodiment of the present invention provides a method for modifying the sharding rules of a distributed database, comprising: in response to detecting a sharding rule modification instruction, determining the original storage node and the new storage node related to the modification of the sharding rule, and the first data table to be modified in the original storage node; obtaining a sharding rule modification parameter, wherein the sharding rule modification parameter is used to indicate the manner in which the distributed database modifies the sharding rules online; when it is determined that the sharding rule modification parameter is a modification between storage nodes, creating a second data table with the same structure as the first data table in the new storage node; inserting the data in the first data table into the second data table through an insert statement, so as to realize data migration between the original storage node and the new storage node; and exchanging the storage paths and sharding rules of the first data table and the second data table in the system table to realize online sharding rule modification of the distributed database.

[0030] Figure 1 An exemplary system architecture to which the method and apparatus for modifying sharding rules of a distributed database of the present invention can be applied is shown.

[0031] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first server 101 , a second server 102 , a third server 103 and a database operating system 104 .

[0033] The first server 101, the second server 102, and the third server 103 may be computing nodes for executing tasks, or storage nodes for storing data in a distributed database. The database operating system 104 may include a server cluster for unified management of multiple databases and servers.

[0034] The network is used as a medium for providing communication links between the first server 101, the second server 102, the third server 103 and the database operating system 104. The network may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0035] It should be noted that the distributed database sharding rule modification method provided in the embodiment of the present invention can generally be executed by the database operating system 104. Accordingly, the distributed database sharding rule modification device provided in the embodiment of the present invention can generally be set in the hardware device carrying the database operating system 104.

[0036] Alternatively, the distributed database sharding rule modification method provided in the embodiment of the present invention can also be executed by the computing nodes in the first server 101, the second server 102, and the third server 103. Correspondingly, the distributed database sharding rule modification device can also be set in the computing nodes in the first server 101, the second server 102, and the third server 103.

[0037] Figure 2 A flow chart of a method for modifying sharding rules in a distributed database according to an embodiment of the present invention is shown.

[0038] like Figure 2 As shown, the embodiment 200 includes operations S210 to S250.

[0039] In operation S210, in response to detecting a sharding rule modification instruction, an original storage node and a new storage node related to the modified sharding rule, and a first data table to be modified in the original storage node are determined.

[0040] In operation S220, a sharding rule modification parameter is obtained, where the sharding rule modification parameter is used to indicate a method for modifying the sharding rule online in the distributed database.

[0041] In operation S230, when it is determined that the sharding rule modification parameter is modification between storage nodes, a second data table having the same structure as the first data table is created in the new storage node.

[0042] In operation S240, the data in the first data table is inserted into the second data table through an insert statement, so that data migration is achieved between the original storage node and the new storage node.

[0043] In operation S250, storage paths and sharding rules of the first data table and the second data table are exchanged in the system table to implement online sharding rule modification of the distributed database.

[0044] According to an embodiment of the present invention, a distributed database may include multiple nodes, and different nodes carry different database tasks. For example, a node used to store a database in a distributed database is a storage node, and a node used to perform computing tasks in a distributed database is a computing node. The computing node may also be called an execution node, a coordination node, a coordinator, etc.

[0045] In a distributed database, computing nodes can interact with storage nodes to obtain corresponding data from the storage nodes.

[0046] The sharding rule modification instruction is used to modify the sharding rules in the distributed database. While modifying the sharding rules, it is also necessary to adapt to the modification of the sharding rules and migrate the data affected by the modification from the original storage node to the new storage node. In the embodiment of the present invention, the sharding rule modification process generally does not involve changes in the number of storage nodes.

[0047] For example, according to the old sharding rules, data A is stored in storage node 1, and data B is stored in storage node 2. According to the new sharding rules, data A is stored in storage node 2, and data B is stored in storage node 1. Therefore, for data A, storage node 1 is called the original storage node, and storage node 2 is called the new storage node; for data B, storage node 2 is called the original storage node, and storage node 1 is called the new storage node.

[0048] Modifying the sharding rules may cause changes in the storage nodes of some data, or may not affect the storage nodes of other data. Therefore, the data table affected by the modification of the sharding rules can be called the first data table to be modified, the storage node to which the first data table belongs before the modification of the sharding rules is called the original storage node, and the storage node after the modification of the sharding rules is called the new storage node.

[0049] According to an embodiment of the present invention, the sharding rule modification instruction may include a modification method of the sharding rule. For example, the original sharding rule is modified to a new sharding rule, or only the modified sharding rule is included. The sharding rule includes the modification of the sharding method and the sharding key, and the sharding key is also a field in the data table.

[0050] Therefore, in response to detecting a sharding rule modification instruction, the original storage node, the new storage node, and the first data table to be modified that are related to the modified sharding rule can be determined based on the sharding rule modification instruction.

[0051] The sharding rule modification instruction may be a SQL statement written in Structured Query Language (SQL).

[0052] For example, the sharding rule modification instruction may be ALTER TABLE table_name DISTRIBUTE BY shard(column_name [, column_name]). The sharding rule modification instruction includes the first data table table_name, the modified new sharding rule shard(column_name [, column_name]), shard is the sharding mode, column_name and [, column_name] respectively represent one or more columns in the new sharding rule, and when the sharding mode is a replicated table, (column_name [, column_name]) may be empty. The modified sharding mode may include hash sharding, range sharding, or replicated table.

[0053] The embodiments of the present invention support multiple online modification modes of sharding rules, which can be reflected through sharding rule modification parameters.

[0054] For example, the sharding rule modification parameters include inter-storage node modification. Inter-storage node modification means that when modifying the sharding rule, the data in the first data table is only migrated between the original storage node at the bottom layer and the new storage node, and there is no need to pull the data in the first data table to the upper layer computing node for caching.

[0055] Therefore, when it is determined that the sharding rule modification parameter is a modification between storage nodes, data migration between the original storage node and the new storage node can be achieved by table-to-table migration.

[0056] Specifically, since the modification of the sharding rules only involves changes in the storage location and generally does not involve changes in the data structure, when it is determined that the modification parameters of the sharding rules are modifications between storage nodes, a second data table with the same structure as the first data table is created at the new storage node to ensure that the data in the first data table can be stored in the second data table according to the same data structure.

[0057] It should be noted that the table structure here is the data structure, such as integer, string, etc., but the definition statement of the second data table is created based on the new sharding rule. The definition statement is also a table creation statement written based on the Data Definition Language (DDL).

[0058] For example, since the first data table was created before the sharding rule was modified, the definition statement of the first data table was written based on the original sharding rule, and its definition statement was CREATE TABLE TableA(a int, b int) DISTRIBUTE BYhash(a), that is, the original sharding rule used the a field as the sharding key.

[0059] After detecting the sharding rule modification instruction ALTER TABLE TableA DISTRIBUTE BY hash(b), the definition statement for creating the second data table is CREATE TABLE TableB(a int, b int) DISTRIBUTE BY hash(b), which means that the new sharding rule uses the b field as the sharding key. In addition, it can be seen that the data structure in the first data table TableA and the second data table TableB is still integer (int), and only the sharding key has changed.

[0060] According to an embodiment of the present invention, the insert statement may also refer to an SQL statement written based on the Structured Query Language (SQL), so that the computing node controls the original storage node and the new storage node based on the insert statement.

[0061] In an embodiment of the present invention, the insert statement is used to insert data in the first data table into the second data table, so as to achieve data migration between the original storage node and the new storage node.

[0062] According to an embodiment of the present invention, after all the data in the first data table is inserted into the second data table, at this time, there are two addresses storing the same data at the same time. In order to ensure that the user's query operation can adapt to the new sharding rule, that is, the query operation under the new sharding rule can query the part of data at the correct location, it is necessary to modify the index information of the part of data.

[0063] In an embodiment of the present invention, the system table is a data table storing data index information, which facilitates the computing node to determine the corresponding storage node based on the index information in the system table. Therefore, after inserting the data of the first data table into the new storage node, the storage path of the first data table and the second data table are exchanged in the system table, so that after modifying the shard information, the query operation can query the part of the data at the correct location.

[0064] In addition to exchanging the storage paths of the first data table and the second data table in the system table, the original sharding rule of the first data table needs to be modified to a new sharding rule. For example, the sharding rule in the system table can be modified by exchanging the sharding rule, as in the above storage path modification method.

[0065] For example, the storage path of the first data table TableA points to / data / 100, and the sharding rule is hash(a), and the storage path of the second data table TableB points to / data / 200, and the sharding rule is hash(b). In the modified system table, the storage path of the first data table TableA points to / data / 200, and the sharding rule is hash(b), and the storage path of the second data table TableB points to / data / 100, and the sharding rule is hash(a).

[0066] The embodiments of the present invention support multiple online sharding rule modifications through sharding rule modification parameters, which facilitates the flexible implementation of online sharding rules. In addition, based on the modification method between storage nodes, it supports direct data migration between the original storage node and the new storage node at the bottom layer, without pulling the data to the upper computing node, which has higher modification efficiency and faster modification speed. Since there is no need to occupy the data pulled from the original storage node in the computing node cache, the embodiments of the present invention can also occupy less computing resources.

[0067] According to an embodiment of the present invention, an insert statement includes a query substatement and an insert substatement, and a first data table includes at least one data record. Inserting data from the first data table into a second data table through the insert statement includes: querying each data record from the first data table in real time based on the query substatement through a first thread; and inserting the data record queried from the first data table into the second data table synchronously based on the insert substatement through a second thread parallel to the first thread.

[0068] In an embodiment of the present invention, the query substatement may be a SELECT statement, and the insert substatement may be an INSERT INTO. The query substatement is used to query data from the first data table, that is, read data, and the insert data is used to insert data from the first data table into the second data table.

[0069] It should be noted that the computing node can start two threads based on the insert statement to execute the query operation and the insert operation in parallel on the original storage node and the new storage node.

[0070] According to an embodiment of the present invention, the computing node can query / obtain each data record in the first data table in real time through the first thread, and at the same time, insert each data record queried / obtained from the first data table into the second data table in real time through the second thread.

[0071] The embodiment of the present invention can query / read data in the original storage node and insert data in the new storage node in parallel through a simple SQL statement, so that the execution speed of online sharding rule modification is faster and more efficient, thereby improving the user experience.

[0072] Figure 3A A schematic diagram of a method for modifying sharding rules in the case of modification between storage nodes according to an embodiment of the present invention is shown.

[0073] like Figure 3A As shown, the computing node uses the first thread to query each data record in real time from the first data table of the original storage node based on the query sub-statement; and uses the second thread to insert each queried data record into the second data table of the new storage node in real time based on the insert sub-statement, thereby realizing data migration between the underlying original storage node and the new storage node.

[0074] According to an embodiment of the present invention, the method further includes: after determining the first data table to be modified in the original storage node, adding a write-disable lock for the first data table, so that when the sharding rules are modified, the write operation to the first data table is stopped.

[0075] In order to ensure the consistency of data in the original storage node and the new data node during the modification of the sharding rules, a write-disable lock can be added to the first data table to prevent users or distributed databases from performing write operations on the first data table, thereby avoiding the phenomenon of missing newly added data in the second data table.

[0076] It should be noted that the write-inhibit lock is implemented based on the lock mechanism of the distributed database and is only used to stop or prohibit the write operation to the first data table. The write-inhibit lock does not affect the user's query operation on the data in the first data table.

[0077] In actual application, the data migration speed between storage nodes is related to the amount of data being migrated, that is, the more data there is in the first data table, the slower the data in the first data table is inserted into the second data table. Considering the impact of online sharding rule modification on users, only stopping the write operation by disabling the write lock, but not stopping the user's query operation on the first data table, can minimize the impact on the user side and the business side and improve the user experience.

[0078] According to an embodiment of the present invention, the method further includes: after inserting the data in the first data table into the second data table, increasing the lock level for the first data table, so that when the sharding rules are modified, the write operation and query operation on the first data table are stopped.

[0079] In an embodiment of the present invention, the distributed database includes a locking mechanism for multiple nodes, thereby controlling the level of operations on data in the distributed database. For example, a lower level write-inhibit lock is only used to stop write operations, and a higher level exclusive lock, i.e., a DDL lock, is used not only to stop write operations but also to stop query operations.

[0080] According to an embodiment of the present invention, the method further includes: before raising the lock level for the first data table, in response to detecting a user query instruction for the first data table, querying target data records from the first data table according to query parameters in the query instruction.

[0081] According to an embodiment of the present invention, after adding a write-inhibit lock for the first data table and before raising the lock level for the first data table, the user can query information from the first data table at any time.

[0082] It should be noted that, since the storage path of the data in the system table has not changed at this time and the lock level has not been upgraded, after the computing node detects the user's query instruction for the first data table, it can locate the first data table from the system table according to the query parameters in the query instruction, and query the target data records from the first data table.

[0083] The most time-consuming operation in online sharding rule modification is generally the operation of inserting data from the first data table into the second data table. Other operations only involve the modification of some parameters and the modification speed is relatively fast. Therefore, before inserting data from the first data table into the second data table, a write-forbidden lock is added to stop the user's write operation on the first data table, so that user queries are not affected during the more time-consuming stage; after the insertion operation is completed, the lock level is increased, such as upgrading the write-forbidden lock to a DDL lock, to ensure the correctness of the query data and minimize the impact on user queries.

[0084] According to an embodiment of the present invention, the method further includes: after exchanging the storage paths of the first data table and the second data table in the system table, deleting the first data table in the original storage node; and releasing the lock on the first data table.

[0085] Since the data migration is completed in operation S240 and the storage path exchange is completed in operation S250, it can be considered that the modification of the sharding rule for the first data table in the distributed database has been completed, and the user can perform data writing and query operations normally. Therefore, after operation S250, the first data table is deleted in the original storage node to release the storage resources in the original storage node; at the same time, the DDL lock for the first data table is released so that the user can perform data writing and query operations normally.

[0086] It should be noted that since the write-disable lock and DDL lock are both for the first data table, and the system table modification is achieved by exchanging the storage path, after operation S250, the write-disable lock and DDL lock for the first data table are actually converted into the write-disable lock and DDL lock for the second data table. That is, after operation S250, the first data table and the second data table are exchanged, the first data table (the original table) is deleted, and the lock for the first data table (the new table) is released.

[0087] According to an embodiment of the present invention, exchanging the storage paths of the first data table and the second data table in the system table includes: exchanging the physical file name of the first data table and the physical file name of the second data table in the system table. The physical file name, that is, the name of the file that actually stores the data, can be the name of the specific corresponding file on the disk.

[0088] The following 9 steps will briefly describe the modification logic of the modification method between storage nodes:

[0089] S1. Start a transaction.

[0090] S2. Add a write-disable lock to the first data table, which may be Table A. The lock level of the write-disable lock allows the user to query the first data table, but cannot perform write operations such as modification.

[0091] S3. Create a second data table, TableB, on the new storage node with the same table structure as the first data table TableA.

[0092] S4. Insert all the data in the first data table TableA into the second data table TableB through an insert statement (INSERT INTO TableB SELECT * FROM TableA).

[0093] S5. Increase the lock level of the first data table TableA, and prohibit query operations and write operations on the first data table TableA.

[0094] S6. Exchange the physical file names of the first data table TableA and the second data table TableB stored in the system table.

[0095] S7, delete the second data table TableB. Since the physical file names are exchanged, the data in the storage path of the second data table TableB is actually the data in the first data table TableA. Therefore, the second data table TableB is deleted in the form of SQL statement implementation, and the actual content is to delete the data in the first data table TableA.

[0096] S8. Transaction commit.

[0097] S9. Unlock the table.

[0098] According to an embodiment of the present invention, the method also includes: when it is determined that the sharding rule modification parameters are modified interactively between the computing node and the storage node, storing the data in the first data table to the computing node; deleting the data in the first data table; in the system table, modifying the original sharding rule of the first data table to a new sharding rule; and storing the data in the first data table stored in the computing node to the new storage node according to the new sharding rule.

[0099] Figure 3B A schematic diagram of a sharding rule modification method in the case of interactive modification of computing nodes and storage nodes according to an embodiment of the present invention is shown.

[0100] like Figure 3B As shown, the interactive modification method of the computing node and the storage node is that the computing node pulls all the data in the first data table and stores it in the cache of the computing node, and deletes the data in the first data table. Then, after modifying the original sharding rule of the first data table to a new sharding rule in the system table, the data in the first data table stored by the computing node is stored in the new storage node based on the new sharding rule in the system table.

[0101] According to an embodiment of the present invention, before storing the data in the first data table to the computing node, it is necessary to add a DDL lock to the first data table to stop query operations and write operations to the table.

[0102] The following will briefly describe the modification logic of the interaction modification method between computing nodes and storage nodes through 8 steps:

[0103] S1. Start a transaction.

[0104] S2. Add a DDL lock to the first data table. The lock level of the DDL lock does not allow the user to perform query operations and write operations such as modification on the first data table.

[0105] S3. Query all data from the first data table at the computing node and save the data to the computing node.

[0106] S4. Delete all data in the first data table.

[0107] S5. Modify the sharding rules of the table stored in the system table.

[0108] S6: insert all data in the first data table stored in the computing node into the new storage node according to the new sharding rule, for example, into the second data table in the new storage node.

[0109] S7. Transaction commit.

[0110] S8. Unlock the table.

[0111] The embodiments of the present invention support direct data migration between the original storage nodes and the new storage nodes at the bottom layer based on the modification method between storage nodes, without pulling the data to the upper computing nodes, and the modification efficiency is higher and the modification speed is faster. Since there is no need to cache the data pulled from the original storage node in the computing node, the embodiments of the present invention can also occupy fewer computing resources. In addition, by parallelizing the first thread and the second thread, the modification speed and modification efficiency can be further improved. Adding a write-disable lock and increasing the lock level can reduce the impact on the user side and the business side and improve the user experience. Supporting multiple online sharding rule modifications through sharding rule modification parameters facilitates the flexible implementation of online sharding rules.

[0112] Figure 4 A block diagram of an apparatus for modifying sharding rules of a distributed database according to an embodiment of the present invention is shown.

[0113] like Figure 4 As shown, the device 400 for modifying sharding rules of a distributed database includes a determination module 410, an acquisition module 420, a table creation module 430, a data migration module 440 and a path modification module 450.

[0114] The determination module 410 is used to determine the original storage node and the new storage node related to the modified sharding rule, and the first data table to be modified in the original storage node in response to detecting the sharding rule modification instruction.

[0115] The acquisition module 420 is used to obtain sharding rule modification parameters, wherein the sharding rule modification parameters are used to indicate the manner in which the distributed database modifies sharding rules online.

[0116] The table creation module 430 is used to create a second data table with the same structure as the first data table in a new storage node when it is determined that the sharding rule modification parameter is modified between storage nodes.

[0117] The data migration module 440 is used to insert the data in the first data table into the second data table through an insert statement, so as to realize data migration between the original storage node and the new storage node.

[0118] The path modification module 450 is used to exchange the storage paths and sharding rules of the first data table and the second data table in the system table to implement online sharding rule modification of the distributed database.

[0119] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0120] For example, any multiple of the determination module 410, the acquisition module 420, the table building module 430, the data migration module 440, and the path modification module 450 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the determination module 410, the acquisition module 420, the table building module 430, the data migration module 440, and the path modification module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the determination module 410, the acquisition module 420, the table creation module 430, the data migration module 440 and the path modification module 450 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0121] It should be noted that the device part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention. The description of the device part specifically refers to the method part, which will not be repeated here.

[0122] Figure 5A block diagram of an electronic device applicable to a method for modifying sharding rules of a distributed database according to an embodiment of the present invention is shown. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0123] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0124] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0125] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0126] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0127] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0128] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0129] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0130] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0131] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0132] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0133] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0135] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for modifying sharding rules of a distributed database, characterized in that: include: In response to detecting a sharding rule modification instruction, determining an original storage node and a new storage node related to modifying the sharding rule, and a first data table to be modified in the original storage node; Obtaining a sharding rule modification parameter, wherein the sharding rule modification parameter is used to indicate a method for modifying a sharding rule online in a distributed database; When it is determined that the sharding rule modification parameter is modification between storage nodes, creating a second data table having the same structure as the first data table at the new storage node; Inserting the data in the first data table into the second data table through an insert statement, so as to achieve data migration between the original storage node and the new storage node; and The storage paths and sharding rules of the first data table and the second data table are exchanged in the system table to implement online sharding rule modification of the distributed database.

2. The method according to claim 1, characterized in that The insert statement includes a query sub-statement and an insert sub-statement, and the first data table includes at least one data record; Inserting the data in the first data table into the second data table by using an insert statement includes: Through the first thread, query each data record in real time from the first data table based on the query sub-statement; The data records queried from the first data table are synchronously inserted into the second data table based on the insert sub-statement through a second thread running in parallel with the first thread.

3. The method according to claim 1, characterized in that The method further comprises: After determining the first data table to be modified in the original storage node, a write-disable lock is added for the first data table, so that when the sharding rule is modified, the write operation to the first data table is stopped.

4. The method according to claim 3, characterized in that The method further comprises: After the data in the first data table is inserted into the second data table, the lock level for the first data table is increased, so that when the sharding rule is modified, the write operation and the query operation on the first data table are stopped.

5. The method according to claim 4, characterized in that The method further comprises: Before raising the lock level for the first data table, in response to detecting a query instruction from a user for the first data table, querying target data records from the first data table according to query parameters in the query instruction.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: After exchanging the storage paths of the first data table and the second data table in the system table, deleting the first data table in the original storage node; The lock on the first data table is released.

7. The method according to claim 1, characterized in that The exchanging the storage paths and sharding rules of the first data table and the second data table in the system table includes: In the system table, the physical file name and the fragmentation rule of the first data table are exchanged with the physical file name and the fragmentation rule of the second data table.

8. The method according to claim 1, characterized in that The method further comprises: When it is determined that the sharding rule modification parameter is modified interactively by a computing node and a storage node, storing the data in the first data table in the computing node; Deleting the data in the first data table; In the system table, modifying the original sharding rule of the first data table into a new sharding rule; and According to the new sharding rule, the data in the first data table stored by the computing node is stored in the new storage node.

9. A device for modifying sharding rules of a distributed database, characterized in that: include: A determination module, configured to determine, in response to detecting a sharding rule modification instruction, an original storage node and a new storage node related to modifying the sharding rule, and a first data table to be modified in the original storage node; An acquisition module, used to acquire a sharding rule modification parameter, wherein the sharding rule modification parameter is used to indicate a method for modifying a sharding rule online in a distributed database; A table creation module, configured to create a second data table having the same structure as the first data table at the new storage node when determining that the sharding rule modification parameter is modified between storage nodes; a data migration module, configured to insert the data in the first data table into the second data table through an insert statement, so as to achieve data migration between the original storage node and the new storage node; and The path modification module is used to exchange the storage paths and sharding rules of the first data table and the second data table in the system table to realize online sharding rule modification of the distributed database.

10. An electronic device comprising: one or more processors; a memory for storing one or more programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

Images