Method, device and node for managing distributed database, and distributed database
By recording the clear command identifier information in the distributed database and comparing the versions, the problem of inconsistent data table states caused by different node receiving delays is solved, ensuring that the operation is executed in the correct version of the data table and improving the accuracy of the operation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING OCEANBASE TECHNOLOGY CO LTD
- Filing Date
- 2022-10-26
- Publication Date
- 2026-07-21
Smart Images
Figure CN115658812B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of distributed database technology, and in particular to a distributed database management method, apparatus, node, and distributed database. Background Technology
[0002] Distributed databases typically consist of multiple nodes. Data in the same table can be divided into multiple data partitions, and these partitions are stored on different nodes. When a user issues a clear command to empty a table in a distributed database, the different latency at which each node receives the command means that at the same moment, each node may see different states of the table. For example, some nodes may see the table before the clear command, while others may see the table after the clear command. If, within a short period before or after the clear command, a node receives other commands, such as query or update commands, problems will arise in processing these commands. Summary of the Invention
[0003] To overcome the problems existing in related technologies, this specification provides a method, apparatus, node, and distributed database for managing a distributed database.
[0004] According to a first aspect of the embodiments of this specification, a method for managing a distributed database is provided, the method being used for nodes in the distributed database, the method comprising:
[0005] Obtain a command to perform a target operation on a data table to be operated on in the distributed database, wherein the command carries the identification information of the data table to be operated on, as well as version-related information;
[0006] A target data table matching the identification information is determined. If the target data table records a clear command identification information, the target data table is determined to be the data table to be operated based on the comparison result between the clear command identification information and the version-related information. The clear command identification information is recorded after receiving a clear command for the target data table and is used to indicate the version of the target data table after the clear command is executed.
[0007] The command is executed based on the determination result, and the execution result is returned.
[0008] According to a second aspect of the embodiments of this specification, a method for managing a distributed database is provided, the method being applicable to nodes storing data partitions of a target data table, the method comprising:
[0009] Receive a distributed transaction request, which is generated after receiving a clear command for the target data table;
[0010] The distributed transaction is executed in accordance with the distributed transaction request, wherein the distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording clear command identification information in the target data table, and unlocking the target data table, wherein the clear command identification information is used to indicate the version of the target data table after executing the clear command, and when the target data table is in a locked state, update operations on the target data table are not allowed;
[0011] If all operations included in the distributed transaction are executed successfully, the execution result is committed.
[0012] If any operation included in the distributed transaction fails to execute successfully, the distributed transaction is rolled back.
[0013] According to a third aspect of the embodiments of this specification, a method for managing a distributed database is provided, the method comprising:
[0014] Receive the clear command sent by the client to clear the target data table;
[0015] A distributed transaction is created based on the clear command, wherein the distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and unlocking the target data table. The clear command identification information is used to indicate the version of the target data table after executing the clear command. When the target data table is in a locked state, update operations on the target data table are not allowed.
[0016] A distributed transaction request is sent to the node that stores the data partition of the target data table, so that the node that stores the data partition of the target data table executes the distributed transaction.
[0017] According to a fourth aspect of the embodiments of this specification, a management device for a distributed database is provided, the management device being used for nodes in the distributed database, the management device comprising:
[0018] The acquisition module is used to acquire a command to perform a target operation on a data table to be operated on in the distributed database. The command carries the identification information of the data table to be operated on, as well as version-related information.
[0019] The determination module is used to identify a target data table that matches the identification information. If the target data table records a clear command identification information, the module determines whether the target data table is the data table to be operated based on the comparison result between the clear command identification information and the version-related information. The clear command identification information is recorded after receiving a clear command for the target data table and is used to indicate the version of the target data table after executing the clear command.
[0020] The execution module is used to execute the command based on the determination result and return the execution result.
[0021] According to a fifth aspect of the embodiments of this specification, a node for a distributed database is provided, the node comprising:
[0022] Memory is used to store executable computer instructions;
[0023] A processor for implementing the methods mentioned in the first, second, or third aspects above when executing the computer instructions.
[0024] According to a sixth aspect of the embodiments of this specification, a distributed database is provided, the distributed database including the nodes mentioned in the fifth aspect above.
[0025] The beneficial effects of the embodiments in this specification are as follows: After receiving a clear command for a target data table in a distributed database, the clear command identification information is recorded in the target data table to mark the version of the target data table after the clear command is executed. When a user issues other commands to the target data table, version-related information can also be included in the commands. If the node finds the original table of the target data table based on the identification information carried in other commands, it can determine whether the command points to the original table or the new table corresponding to the target data table based on the comparison result between the version-related information carried in the command and the clear command identification information recorded in the original table. This ensures that the relevant operations can be performed on the correct version of the data table, thereby avoiding the problem of the node performing relevant operations on the wrong version due to the inconsistency between the version of the target data table seen by the node and the version of the target data table pointed to by the command.
[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the embodiments of this specification. Attached Figure Description
[0027] The accompanying drawings, which are incorporated herein by reference and form part of the embodiments thereof, illustrate embodiments consistent with those described herein and, together with the description, serve to explain the principles of those embodiments.
[0028] Figure 1This is a schematic diagram illustrating a distributed database as an exemplary embodiment of this specification;
[0029] Figure 2 A flowchart illustrating a distributed database management method as an exemplary embodiment of this specification;
[0030] Figure 3 This is a schematic diagram illustrating an exemplary embodiment of the present specification of executing commands on a data table in a distributed database;
[0031] Figure 4 This is a schematic diagram illustrating an exemplary embodiment of the present specification of executing a query command on a data table in a distributed database;
[0032] Figure 5 This is a schematic diagram illustrating an exemplary embodiment of the present specification of executing an update command on a data table in a distributed database;
[0033] Figure 6 This is a schematic diagram of the Oceanbase distributed database, illustrating an exemplary embodiment of this specification.
[0034] Figure 7 A logical block diagram of a distributed database management device illustrated in an exemplary embodiment of this specification;
[0035] Figure 8 This is a structural logic block diagram of a node in a distributed database, illustrating an exemplary embodiment of this specification. Detailed Implementation
[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those described in this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments described in this specification as detailed in the appended claims.
[0037] The terminology used in the embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of this specification. The singular forms “a,” “described,” and “the” as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0038] It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0039] Databases typically contain various types of commands for database management. These include commands like the `Truncate Table` command to clear tables, `Query` commands to query tables, and `Update` commands (DML: Data Manipulation Language) commands to add, delete, and modify data. One approach to clearing tables is to delete each row one by one, which is relatively inefficient. To improve efficiency, another approach is to create an empty table with the same name as the original table and replace it. For example, after receiving a clear command, the original table can be deleted, and a new table with the same name can be created. Since deleting and reclaiming the original table are asynchronous, there might be a delay between the deletion and reclamation operations, meaning that both the original and new tables might exist simultaneously in a node.
[0040] For distributed databases, which consist of multiple nodes, when a user issues a clear command to empty a table, the different latency at which each node receives the command means that at any given moment, the state of the table might differ. For example, some nodes might see the table before the clear command, while others might see the table after. In this situation, if a node receives other commands, such as query or update commands, within a short period before or after the clear command, problems may arise in processing these commands.
[0041] For example, such as Figure 1The diagram illustrates a distributed database. A distributed database comprises multiple nodes. Data for the same table is typically divided into multiple data partitions, each stored on a different node. When a client sends a command to clear table 1, this command is usually first sent to a central control service. This central control service can run on one of the nodes and is primarily used for scheduling and managing the distributed database resources. The central control service can then send the command to nodes 1 and 2, which store the data partitions of table 1, so that these two nodes can perform the clear operation. Simultaneously, it can broadcast the command to other nodes (nodes 3 and 4). Since each node maintains a state table to record the status information of each data table, other nodes can update the current state of table 1 based on the received command. Because the clear command is transmitted over the network, the arrival time of the command at different nodes may vary. Therefore, different nodes will not see different table states, preventing errors in processing other commands received by other nodes. For example, node 3 might have already received the broadcast command, so the state of table 1 it sees might be the new table after being cleared. Node 4, however, hasn't received the broadcast command yet, so it might see the original table. In this case, if node 4 receives a query command after receiving the clear command, it might still retrieve the data from the original table, which is clearly incorrect. Alternatively, if it receives an update command (insert, delete, or modify), it will still perform the operations on the original table, potentially causing these operations to be replaced by the newly created empty table and not saved.
[0042] Based on this, the embodiments of this specification provide a method for managing a distributed database. Considering the different latency of each node in the distributed database receiving user commands, after issuing a clear command to a data table (e.g., data table A) in the distributed database, some nodes may see the original table of data table A before the clear command was executed, while other nodes may see the new table after the clear command was executed. Before or after the clear command, nodes may also receive other types of commands for data table A. Some of these commands need to be executed in the original table, and some need to be executed in the new table A.
[0043] To ensure commands are executed on the correct version of the data table and avoid errors, upon receiving a clear command for data table A, a clear command identifier is recorded in the original data table A. This identifier marks the version information of data table A after the clear command is executed. When a user issues other commands to data table A, they can also include the identifier and version information of data table A in the command. If the node finds the original data table A based on the identifier information in the other command, it can compare the version information in the command with the clear command identifier recorded in the original data table A to determine whether the command is intended to process the original or a new table of data table A. This ensures that the relevant operations are performed on the correct data table. This avoids the problem of the node performing operations on the wrong version of data table A due to a discrepancy between the version of data table A seen by the node and the version of data table A to be processed in the command.
[0044] The distributed database management method provided in this disclosure can be used to manage various types of distributed databases, such as Oceanbase databases or other types of distributed databases.
[0045] The distributed database management method in this embodiment can be executed by any node or a designated node in the distributed database. For example, it can be executed by any node that receives a user command from the client, or the node that receives the command can send the command to the node that stores the data table to be operated on, and the node that stores the data table to be operated on can then execute the command.
[0046] like Figure 2 As shown, the method may include the following steps:
[0047] S202. Obtain a command to perform a target operation on the data table to be operated on in the distributed database, wherein the command carries the identification information of the data table to be operated on, as well as version-related information.
[0048] In step S202, a command to perform a target operation on a data table in the distributed database can be obtained. This command carries the identification information of the data table to be operated on, as well as version-related information. Specifically, the command to perform the target operation on the data table can be obtained from a user-inputted client. For example, after the user inputs the command through the client, the client can parse the command and send it to the node executing the method based on a certain strategy (such as a load balancing strategy or pre-stored routing information). Alternatively, the command can be obtained from other nodes; for example, other nodes can obtain the command from the client and then send it to the node executing the method.
[0049] The identification information can be various types of information identifying a specific table in the database, such as the table name or number. Typically, each table in the database has a unique identifier; it's only during the brief process of executing a clear command that two tables might have the same identifier (e.g., table name). Therefore, for this special case, version-related information can be included in the command to distinguish between the tables before and after the clear command. This version-related information can distinguish whether the current command is targeting a table before or after the clear command. For example, it could be the time the command was generated or information indicating the version of the table to be operated on.
[0050] S204. Determine the target data table that matches the identification information. If the target data table records the clear command identification information, determine whether the target data table is the data table to be operated based on the comparison result between the clear command identification information and the version-related information. The clear command identification information is recorded after receiving the clear command for the target data table and is used to indicate the version of the target data table after the clear command is executed.
[0051] In step S204, after obtaining the command to perform the target operation on the data table to be operated on, the target data table matching the identification information carried in the command can be found first. For example, if the identification information is the data table name, then the table with the same name will be found. If there has been no clear command on the target data table in the recent past, then the target data table found based on the identification information is the data table to be operated on, and the target operation can be performed directly on the target data table. Of course, if there has been a clear command on the target data table in the recent past (possibly before or after the command), since the node may not have updated the table's state in time, the target data table found based on the identification information may be the original table or a new table, thus there is a problem of mismatch between the searched target data table and the data table to be operated on.
[0052] Considering that if the original table is retrieved, it might be because the node hasn't updated the target table's state in time, resulting in an incorrect version of the table being displayed, a clear command identifier is recorded in the original table of that table whenever a clear command is received. This identifier indicates the current version of the target table after the clear command is executed. Of course, the version can be represented using various information, such as inferring the version based on the time the clear command was received, or distinguishing versions based on the table's version number.
[0053] Therefore, after locating the target data table based on the identifier information, it's crucial to first confirm whether the target data table contains a clear command identifier. If it does, it indicates that a clear operation is currently being performed on that target data table, suggesting the version of the target data table might be incorrect. Therefore, the version-related information carried in the command can be compared with the clear command identifier recorded in the target data table. Based on the comparison result, it can be determined whether the target data table is the one to be operated on. For example, the version of the data table to be operated on, indicated by the version-related information, can be compared with the version of the target data table, indicated by the clear command identifier, to determine if the versions of the two data tables match.
[0054] S206. Execute the command based on the determination result and return the execution result.
[0055] In step S206, after obtaining the determination result, the command can be executed based on the determination result. For example, the determination result can be used to determine whether the target operation is performed in the target data table or in a new table of the target data table, and the execution result is returned.
[0056] When a clear command is received for a data table, the clear command identifier information can be recorded in the data table to indicate the current version of the data table. Then, other commands issued by the user can also carry version-related information. When the data table is found based on other commands, the comparison results between the clear command identifier information and the version-related information recorded in the data table can be used to determine whether the searched table is the correct table before performing the relevant operations. This can avoid executing commands on the wrong table.
[0057] In some embodiments, if the target data table is determined to be the data table to be operated on based on the comparison result between the version-related information carried in the command and the clear command identification information, then the target operation is performed directly on the target data table.
[0058] In some embodiments, if the target data table is determined not to be operated on based on the comparison result between the version-related information carried in the command and the clear command identification information, then it means that the command points to a new table of the target data table. Here, the new table refers to the data table generated after receiving the clear command to clear the target data table, used to replace the target data table. If the data table to be operated on is a new table, then the new table can be identified as the data table to be operated on, so that the target operation can be performed on the new table.
[0059] For example, such as Figure 3As shown, upon receiving a clear command for data table 1, the clear command identifier information can be recorded in the original table of data table 1, and then a new table is created to replace data table 1. When other commands are received, data table 1 can be searched based on the identifier information carried by the command. If the original table is found, the version information in the command is compared with the clear command identifier information recorded in the original table to determine whether the data table to be operated on is the original table or the new table. If it is the original table, the target operation is performed on the original table; if it is the new table, the target operation is performed on the new table.
[0060] Since database commands typically include query commands and update commands (e.g., DML commands), for query commands, because query operations can be completed quickly, if the query command is issued before the clear command, the node will still see the original table after the clear command, and querying data in the original table will not be a problem. Conversely, if the query command is issued after the clear command, data should be queried in the new table. Therefore, in some embodiments, such as... Figure 4 As shown, the command to perform the target operation on the data table to be operated on can be a query command. Since query commands usually carry timestamp information, this version-related information can be the timestamp corresponding to the query command. In some scenarios, the user can specify to query data in the target data table at a specific point in time; therefore, this timestamp can be the user-specified timestamp. In some scenarios, this timestamp can be the timestamp when the query command was generated. Correspondingly, the clear command identification information can be the timestamp when the clear command was received.
[0061] When determining whether a target data table is the table to be operated on based on the comparison results between the clear command identifier information and version-related information recorded in the target data table, if the timestamp corresponding to the query command is less than the timestamp when the clear command was received, it means that the query command was issued before the clear command was issued. Therefore, the data table pointed to by the query command is the original table before the clear command was executed. Thus, the target data table can be determined as the table to be operated on, and the relevant data can be directly queried and returned from the target data table. If the timestamp corresponding to the query command is equal to or greater than the timestamp when the clear command was received, it means that the query command was issued after the clear command was issued. Therefore, the data table pointed to by the query command is the new table after the clear command was executed. Thus, the new table corresponding to the target data table can be determined as the table to be operated on, and the relevant data can be queried and returned from the new table.
[0062] Of course, for update operations on data tables, such as writing data to a data table, deleting data from a data table, and modifying data in a data table, these operations usually correspond to a process, meaning that executing these commands takes a relatively long time. Therefore, these commands cannot usually be simply considered as commands at a specific point in time; they do not have the concept of a timestamp, and these commands typically do not carry a timestamp. Therefore, for these commands, the table structure version number can be used to identify the table version before and after executing the clear command. For example, each time a clear command is received, the table structure version number changes accordingly, such as version number 1.0, 2.0, etc. Therefore, in some embodiments, such as... Figure 5 As shown, this command can be an update command to perform an update operation on the data table to be operated on, and the version-related information can be the table structure version number of the data table to be operated on. Correspondingly, the clear command identification information can be the table structure version number of the target data table, wherein the table structure version number of the data table will change after the data table is cleared.
[0063] When determining whether a target data table is the table to be operated on based on the comparison results between the clear command identifier information and version-related information recorded in the target data table, if the table structure version number of the table to be operated on is less than the table structure version number of the target data table, then the new table corresponding to the target data table is identified as the table to be operated on. For update commands, the table structure version number can be automatically generated when the command is generated. Generally, there are two situations regarding the table structure version number in the update command: one is that the node executing the method did not receive the clear command in time, therefore, the automatically generated table structure version number in the update command is still the original table structure version number, which will be less than the table structure version number recorded in the target data table. In this case, the new table should be identified as the table to be operated on, and the update operation should be performed directly on the new table. The other is that the clear command has already been received. In this case, the new table is found directly based on the identifier information and the recorded state of the target data table, so the update operation can be performed directly on the new table.
[0064] Furthermore, regarding operations that update data tables, since update operations typically last for a certain period, it's possible that an update operation on the target data table is received before a clear command is received (e.g., inserting data into the target data table previously). If a clear command is received during the data insertion process, creating a new table, it would be more semantically appropriate to continue inserting data into the new table. However, related technologies often execute the clear command within the original table, inserting data there, which results in the loss of the data inserted after the clear command (data that should have been in the new table). To avoid this problem, after receiving a clear command to clear the target data table, the target data table can be locked. This means setting the target data table to a locked state, disallowing update operations while the target data table is locked.
[0065] Therefore, in some embodiments, if the command is an update command that performs an update operation on the target data table, after identifying the target data table based on the identification information, it can be determined whether the target data table is in a locked state. If so, the processing of the update command is paused. This ensures that no add, delete, or modify operations are performed during the process of clearing the target data table, avoiding the loss of data added, deleted, or modified after the clear command.
[0066] In some embodiments, the node can be a node running a central control service. Typically, a distributed database includes a central control service, which runs on a node within the distributed database. This central control service is used for resource scheduling and management within the distributed database. When a user issues a clear command to clear a target data table via a client, the client first sends the clear command to the central control service. Upon receiving the clear command, the central control service can create a distributed transaction based on it. This distributed transaction can include the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and releasing the lock on the target data table. After creating the distributed transaction, a distributed transaction request can be sent to the node storing the data partition of the target data table to instruct the node storing the data partition to execute the aforementioned distributed transaction. The clear command identification information can include one or more of the following: a timestamp when the clear command was received and a table structure version number.
[0067] In some embodiments, the node may be a node storing data partitions of the target data table. After the user issues a clear command to clear the target data table, the node can receive a distributed transaction request issued by the central control service, and then execute the distributed transaction indicated by the distributed transaction request. This distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identifier information in the target data table, and releasing the lock on the target data table. During the execution of the above distributed transaction, if all operations included in the distributed transaction are executed successfully, the execution result is committed. If any operation included in the distributed transaction fails, the distributed transaction is rolled back.
[0068] In some scenarios, when a clear command to clear a target data table is received, there may be other update operations (such as insert, delete, and modify) on the same data table. Since update operations usually take some time, to avoid conflicts between the two commands, before setting the target data table to a locked state, it is possible to first determine whether there are any update operations on the target data table. If so, the update operation can be waited for to complete before setting the target data table to a locked state.
[0069] Furthermore, embodiments of this specification also provide a method for managing a distributed database, applicable to nodes storing data partitions of a target data table, the method comprising:
[0070] Receive a distributed transaction request, which is generated after receiving a clear command for the target data table;
[0071] The distributed transaction is executed in accordance with the distributed transaction request, wherein the distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording clear command identification information in the target data table, creating a new table to replace the target data table, and unlocking the target data table, wherein the clear command identification information is used to indicate the version of the target data table after executing the clear command, and when the target data table is in a locked state, update operations on the target data table are not allowed;
[0072] If all operations included in the distributed transaction are executed successfully, the execution result is committed.
[0073] If any operation included in the distributed transaction fails to execute successfully, the distributed transaction is rolled back.
[0074] In some embodiments, the clear command identification information includes a timestamp when the clear command is received and / or a table structure version number, wherein the table structure version number of the data table will change after a clear command for a data table in a distributed database is received.
[0075] The specific process of this node executing the distributed database management method can be referred to the description in the above embodiments, and will not be repeated here.
[0076] Furthermore, embodiments of this specification also provide a method for managing a distributed database, the method comprising:
[0077] Receive the clear command sent by the client to clear the target data table;
[0078] A distributed transaction is created based on the clear command. The distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and unlocking the target data table. The clear command identification information is used to indicate the version of the target data table after executing the clear command. When the target data table is in a locked state, update operations on the target data table are not allowed.
[0079] A distributed transaction request is sent to the node that stores the data partition of the target data table, so that the node that stores the data partition of the target data table executes the distributed transaction.
[0080] In some embodiments, the clear command identification information includes a timestamp when the clear command is received and / or a table structure version number, wherein the table structure version number of the data table will change after a clear command for a data table in a distributed database is received.
[0081] Typically, distributed databases include a central control service, which can run on one of the nodes of the distributed database. This central control service is used to implement resource scheduling and other functions within the distributed database. When a user issues a clear command to clear a target data table through a client, the client first sends the clear command to the central control service, which then schedules the nodes to process the clear command. Therefore, this distributed database management method can be executed by the node running the central control service.
[0082] The specific process of this node executing the distributed database management method can be referred to the description in the above embodiments, and will not be repeated here.
[0083] To further explain the distributed database management method provided in the embodiments of this specification, the following explanation is based on a specific embodiment.
[0084] like Figure 6 The diagram illustrates the Oceanbase distributed database. Oceanbase typically comprises multiple Zones, each containing a copy of all data stored in the database. Each Zone contains multiple nodes, and each data table is divided into multiple data partitions distributed across these nodes. For example, in Zone 1, data table A includes two data partitions, A1 and A2, stored on nodes 1 and 2 respectively. Node 1 runs a central control service, which manages and schedules resources for the distributed database. When a user issues a clear command to clear data table A through a client, the client forwards the command to the central control service. The central control service then sends the clear command to the nodes storing data table A, namely nodes 1 and 2. Simultaneously, the central control service broadcasts the clear command to other nodes (e.g., node 3), which then update the state of data table A based on the clear command. Because the time delay for the clear command to reach different nodes varies, some nodes may receive the clear command and thus see the cleared data table, while others may not receive the clear command and thus see the data table before the clear command. This leads to two problems: (1) Update commands for updating the data table issued before the clear command, such as data insertion commands, may lose the data that was inserted after the clear command (which should have been inserted into the new table) because the nodes still execute the insertion operation in the original table after receiving the clear command and generating the new table. (2) For query commands or update commands such as insert, delete, and modify that occur after the clear command, the nodes may still execute the query command or the insert, delete, and modify command in the original table because they still see the original table, resulting in errors in the queried data or loss of the data after the insert, delete, and modify.
[0085] Based on this, this embodiment provides a management method for distributed databases. After receiving a command to clear the data in a data table (e.g., data table A), the central control service can create a distributed transaction, which includes the following operations:
[0086] (1) Lock the original table of data table A; after locking, no more add, delete or modify operations are allowed on the original table of data table A. This can prevent add, delete or modify commands received before the clear command from continuing to perform related operations on the original table after the clear command, which would cause data loss.
[0087] (2) Delete data table A;
[0088] (3) Create an empty table with the same name as data table A; since the deleted data table A may not be reclaimed for a period of time after the deletion operation is performed, there is a situation where the original table and the new table coexist.
[0089] (3) Record the current timestamp and table structure version number in the original table of data table A; the table structure version number will change after each clear command is received.
[0090] (4) Unlock data table A.
[0091] After creating a distributed transaction, the central control service sends a distributed transaction request to the node storing the data partition of table A. Upon receiving the distributed transaction request, the node storing the data partition of table A will execute the four operations mentioned above and then submit the execution result to the central control service. The request is considered successful only if all four operations are executed successfully. If any operation fails, the four operations will be rolled back.
[0092] When a user issues a query command for database table A (this command can be issued before or after the clear command), the client can send the query command to a node in the distributed database (e.g., any one of node 1-node 3). The query command can carry the identifier information of table A (e.g., table name) and a timestamp. This timestamp can be a user-specified timestamp (e.g., the user wants to query a specific piece of data in table A at a specific time), or it can be a timestamp automatically generated when the query command was generated. The node receiving the query command (e.g., node 3) can search for table A based on the identifier information carried in the query command. If the original table, which is still table A, is found, the timestamp of the record in the original table can be compared with the timestamp carried in the query command. If the timestamp carried in the query command is less than the timestamp of the record in the original table (meaning the query command was issued before the clear command), then the data can be retrieved from the original table and returned. If the timestamp carried in the query command is greater than or equal to the timestamp of the record in the original table, it means that the query command was issued after the clear command. In this case, the query command can be routed to the new table, and data can be retrieved from the new table and returned.
[0093] When a user issues an update command for database table A (e.g., an insert, delete, or update command, which can be before or after a clear command), the client can send the update command to a node in the distributed database (e.g., any one of node 1-node 3). Since update commands typically take a long time to execute, there are scenarios where a clear command is received halfway through the update command's execution. Therefore, update commands usually do not include timestamps, and the order of commands cannot be compared using timestamp information. Therefore, the update command can carry the identifier information of table A (e.g., the table name) and the table structure version number of the table to be updated. After receiving the update command, the node can find table A based on the identifier information. If the found table is still the original table, it can compare the table structure version number recorded in the original table with the table structure version number carried in the update command. If the table structure version number carried in the update command is less than the table structure version number recorded in the original table, the update command can be routed to the new table, and the update operation can be performed on the new table.
[0094] In addition, if an update command is received, and the original table of data table A is found based on the update command, and the original table is in a locked state at this time, the update operation is paused.
[0095] Corresponding to the distributed database management method embodiments provided in this specification, this specification also provides a distributed database management device, which is used for nodes in a distributed database, such as... Figure 7 As shown, the device 70 includes:
[0096] The acquisition module 72 is used to acquire a command to perform a target operation on a data table to be operated on in the distributed database. The command carries the identification information of the data table to be operated on, as well as version-related information.
[0097] The determination module 74 is used to determine the target data table that matches the identification information. If the target data table records the clear command identification information, the module determines whether the target data table is the data table to be operated based on the comparison result between the clear command identification information and the version-related information. The clear command identification information is recorded after receiving the clear command for the target data table and is used to indicate the version of the target data table after the clear command is executed.
[0098] The execution module 76 is used to execute the command based on the determination result and return the execution result.
[0099] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0100] Furthermore, embodiments of this specification provide a node in a distributed database, which can be a server or a server cluster. For example... Figure 8 As shown, this node, in addition to Figure 8 The processor 82 and memory 84 shown may also include other hardware.
[0101] The memory 84 stores computer instructions for execution by the processor 82. When the processor 82 executes the computer instructions, it implements the distributed database management method provided in all or part of the above embodiments.
[0102] Furthermore, embodiments of this specification provide a distributed database, which includes multiple nodes, which may be servers or server clusters, and these nodes may be used to implement the distributed database management methods in some or all of the above embodiments.
[0103] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments in this specification, depending on actual needs. Those skilled in the art can understand and implement this without creative effort.
[0104] Since the parts of the embodiments in this specification that contribute to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes several instructions to cause a terminal device to execute all or part of the steps of the methods in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0105] The above description is merely a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this specification should be included within the scope of protection of the embodiments of this specification.
Claims
1. A method for managing a distributed database, the method being used for nodes in a distributed database, the method comprising: Obtain a command to perform a target operation on a data table to be operated on in the distributed database. The command carries the table identifier information of the data table to be operated on, as well as version-related information. The table identifier information is used to indicate the matching data table. A target data table matching the table identifier information is identified. If the target data table records a clear command identifier, based on the comparison result between the clear command identifier and the version-related information, it is determined whether the target data table or the new table corresponding to the target data table is the data table to be operated on. The clear command identifier is recorded in the target data table while retaining it after receiving a clear command for the target data table. The clear command identifier is used to associate the target data table with the new table generated after executing the clear command. The new table is used to replace the target data table. The command is executed based on the determination result, and the execution result is returned.
2. The method according to claim 1, wherein executing the command based on the determination result and returning the execution result includes: If the target data table is determined to be the data table to be operated on, then the target operation is performed on the target data table; If it is determined that the target data table is not the data table to be operated on, then the new table corresponding to the target data table is determined as the data table to be operated on, and the target operation is performed in the new table, wherein the new table is a data table generated after receiving the clear command to replace the target data table.
3. The method according to claim 1 or 2, wherein the command includes a query command to perform a query operation on the data table to be operated on, the version-related information includes the timestamp corresponding to the query command, and the clear command identification information includes the timestamp when the clear command is received; Based on the comparison result between the clear command identifier information and the version-related information, determining whether the target data table is the data table to be operated on includes: If the timestamp corresponding to the query command is less than the timestamp when the clear command is received, the target data table is determined to be the data table to be operated on. If the timestamp corresponding to the query command is not less than the timestamp when the clear command is received, the new table corresponding to the target data table is determined as the data table to be operated on.
4. The method according to claim 1 or 2, wherein the command includes an update command to perform an update operation on the data table to be operated on, the version-related information includes the table structure version number of the data table to be operated on, and the clear command identification information includes the table structure version number of the target data table, wherein, After receiving a clear command for a data table in a distributed database, the table structure version number will change. The step of determining whether the target data table is the data table to be operated on based on the comparison result between the clear command identifier information and the version-related information includes: If the table structure version number of the data table to be operated on is less than the table structure version number of the target data table, the new table corresponding to the target data table is determined as the data table to be operated on.
5. The method according to claim 1, wherein the command includes an update command for performing an update operation on the data table to be operated on, and the method further includes: If the target data table is determined to be in a locked state, the processing of the update command is suspended; wherein, after receiving a clear command for the target data table, the target data table is set to a locked state.
6. The method according to claim 1, wherein the node is a node storing a data partition of the target data table, and the method further comprises: Receive distributed transaction requests; Execute the distributed transaction indicated by the distributed transaction request, wherein the distributed transaction includes the following operations executed in sequence: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and unlocking the target data table; If all operations included in the distributed transaction are executed successfully, the execution result is committed. If any operation included in the distributed transaction fails to execute successfully, the distributed transaction is rolled back.
7. The method according to claim 6, further comprising, before setting the target data table to a locked state: Determine whether there is currently an update operation for the target data table; If it exists, the target data table will be set to a locked state after the update operation is completed.
8. The method according to claim 1, further comprising: Receive the clear command sent by the client to clear the target data table; A distributed transaction is created based on the clear command, wherein the distributed transaction includes the following operations executed in sequence: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and unlocking the target data table. A distributed transaction request is sent to the node that stores the data partition of the target data table, so that the node that stores the data partition of the target data table executes the distributed transaction.
9. A method for managing a distributed database, the method being applicable to nodes storing data partitions of a target data table, the method comprising: Receive a distributed transaction request, which is generated after receiving a clear command for the target data table; The distributed transaction is executed according to the distributed transaction request instruction, wherein the distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording clear command identification information in the target data table, and unlocking the target data table. The clear command identification information is recorded in the target data table while retaining it after receiving a clear command for the target data table. The clear command identification information is used to associate the target data table with the new table generated after executing the clear command. The new table is used to replace the target data table. When the target data table is in a locked state, update operations on the target data table are not allowed. If all operations included in the distributed transaction are executed successfully, the execution result is committed. If any operation included in the distributed transaction fails to execute successfully, the distributed transaction is rolled back.
10. The method according to claim 9, wherein the clear command identification information includes a timestamp when the clear command is received, and / or a table structure version number, wherein, When a clear command is received for a data table in a distributed database, the table structure version number will change.
11. A method for managing a distributed database, the method comprising: Receive the clear command sent by the client to clear the target data table; A distributed transaction is created based on the clear command, wherein the distributed transaction includes the following operations executed sequentially: setting the target data table to a locked state, deleting the target data table, creating a new table to replace the target data table, recording the clear command identification information in the target data table, and unlocking the target data table. The clear command identification information is recorded in the target data table while retaining it after receiving the clear command. The clear command identification information is used to associate the target data table with the new table generated after executing the clear command. The new table is used to replace the target data table. When the target data table is in a locked state, update operations on the target data table are not allowed. A distributed transaction request is sent to the node that stores the data partition of the target data table, so that the node that stores the data partition of the target data table executes the distributed transaction.
12. The method according to claim 11, wherein the clear command identification information includes a timestamp when the clear command is received, and / or a table structure version number, wherein, When a clear command is received for a data table in a distributed database, the table structure version number will change.
13. A management device for a distributed database, the management device being used for nodes in the distributed database, the management device comprising: The acquisition module is used to acquire a command to perform a target operation on a data table to be operated on in the distributed database. The command carries table identification information of the data table to be operated on, as well as version-related information. The table identification information is used to indicate the matching data table. The determination module is used to identify a target data table that matches the table identification information. If the target data table contains a clear command identification information, the module determines whether the target data table or a new table corresponding to the target data table is the data table to be operated on, based on a comparison between the clear command identification information and the version-related information. The clear command identification information is recorded in the target data table while retaining the target data table after receiving a clear command. The clear command identification information is used to associate the target data table with the new table generated after executing the clear command. The new table is used to replace the target data table. The execution module is used to execute the command based on the determination result and return the execution result.
14. A node in a distributed database, the node comprising a processor and a memory, the memory storing a computer program, wherein the processor, when executing the computer program, implements the method of any one of claims 1-8, 9-10, or 11-12.
15. A distributed database, the distributed database comprising the nodes as described in claim 14.