Method and system for guaranteeing distributed database DDL operation atomicity based on WAL

Abstract

The invention discloses a method and system for guaranteeing distributed database DDL operation atomicity based on a WAL, and relates to the technical field of distributed databases. The method comprises the steps that a two-stage submission mode is used for executing DDL statements, metadata are modified in the first stage and marked as a specified intermediate state, transaction submission is conducted in the second stage, and the transaction submission is conducted in the second stage; before transaction submission, a synchronous execution job is created, and the job comprises a job ID serving as a unique identifier and metadata of an object involved in the DDL operation; when the job is executed, the database system sends the related distributed operation to the corresponding engine according to the DDL operation type; in the process of executing the job, when execution of a certain node fails or abnormal downtime occurs, all nodes are driven to perform rollback operation; after all the nodes successfully execute the DDL operation, all the nodes are driven to conduct commit operation, and final submission of the DDL operation is completed. According to the method, the ordered execution and reliability of the operation can be ensured, and the atomicity and consistency of the DDL operation are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed databases, and in particular to a method and system for ensuring the atomicity of DDL operations of distributed databases based on WAL. Background Art

[0002] Distributed databases distribute data across multiple nodes and provide services together through network connections to improve the scalability, availability, and performance of the system. Its architecture includes data sharding and replication to ensure load balancing and fault tolerance, and supports parallel processing and fault recovery. Distributed databases make trade-offs between consistency, availability, and partition tolerance (CAP theorem), and adopt various consistency models such as strong consistency and eventual consistency. Despite the complexity of management, they are widely used in modern application scenarios that need to process large amounts of data and high concurrency, such as Internet services, big data analysis, and the Internet of Things.

[0003] WAL (Write-Ahead Logging) is a common logging mechanism in database management systems to ensure data persistence and consistency. In WAL mode, all data modification operations are first written to a write-ahead log file (Write-Ahead Log) and then applied to the actual data file. The advantage of this is that even in the event of a system crash or unexpected power outage, data can be restored by replaying the log file, thereby ensuring data integrity and consistency.

[0004] Atomicity: Atomicity is one of the ACID characteristics of database transactions, ensuring that all operations in a transaction are either completely successful or completely failed. If a transaction encounters an error during execution, all executed operations will be rolled back and restored to the state before the transaction started, thereby maintaining data consistency.

[0005] In a distributed database system, it is crucial to ensure the atomicity of DDL (data definition language) operations. The main reasons include:

[0006] 1. Data consistency: Ensure that DDL operations are either completely successful or completely failed to prevent partial execution from causing data inconsistency or system metadata corruption.

[0007] 2. System stability: Atomic DDL operations avoid system instability when execution is interrupted or failed, ensuring the reliability of the database architecture.

[0008] 3. Concurrency control: In a multi-node environment, atomic DDL operations prevent conflicts caused by concurrent modifications and ensure that all nodes have a consistent view of database structure changes.

[0009] 4. Fault recovery: Atomicity ensures that DDL operations can be correctly rolled back or recovered when the system fails or crashes, ensuring that the system can reliably return to a stable state.

[0010] 5. Simplified operations: Simplify transaction management and troubleshooting, improve development and operation efficiency, and reduce complexity.

[0011] The atomicity of DDL operations is extremely important in distributed databases. It ensures data consistency and system reliability, and improves database stability and ease of operation. However, for distributed multi-mode databases, not all engines can support transaction consistency processing. Summary of the invention

[0012] In view of the needs and shortcomings of current technology development, the present invention provides a method and system for ensuring the atomicity of DDL operations in a distributed database based on WAL. Through a two-phase commit protocol and WAL log records, it is ensured that the atomicity and consistency of DDL operations can be guaranteed even in a distributed environment.

[0013] In the first aspect, the present invention provides a method for ensuring the atomicity of DDL operations in a distributed database based on WAL. The technical solution adopted to solve the above technical problems is as follows:

[0014] A method and system for ensuring the atomicity of DDL operations in a distributed database based on WAL, comprising the following steps:

[0015] S1. The database system executes DDL statements using a two-phase commit method. In the first phase, the metadata is modified and marked as a specified intermediate state. In the second phase, the transaction is committed. Before the transaction is committed in the second phase, the database system creates a synchronously executed job. The job contains a jobID as a unique identifier and the metadata of the objects involved in this DDL operation.

[0016] S2. When executing a job, the database system sends the relevant distributed operations to the corresponding engine according to the DDL operation type;

[0017] S3: When a node fails to execute or crashes during the job execution, all nodes are driven to perform rollback operations;

[0018] S4. After all nodes successfully execute the DDL operation, the upper layer of the database system drives all nodes to perform the commit operation to complete the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

[0019] Optionally, step S1 is performed, and the database system executes the DDL statement using a two-phase commit method, specifically including:

[0020] For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation.

[0021] For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

[0022] Optionally, executing step S2, the engine needs to first record a WAL log entry containing the jobID before modifying the table structure.

[0023] Optionally, execute step S3. When the engine execution of a node fails due to foreseeable reasons, the database system will trigger a rollback operation. All nodes participate in the rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback.

[0024] For nodes that cannot return error information due to abnormal downtime, the database system triggers a rollback operation after a predetermined timeout period has passed, and all nodes participate in the rollback process.

[0025] Further optionally, step S4 is executed. After the WAL modules of all nodes complete the commit operation, the upper layer of the database system will complete the metadata modification of this DDL and change the intermediate state of the object to the public state.

[0026] In a second aspect, the present invention provides a system for ensuring the atomicity of DDL operations in a distributed database based on WAL. The technical solution adopted to solve the above technical problems is as follows:

[0027] A system for ensuring the atomicity of DDL operations in a distributed database based on WAL, comprising:

[0028] The execution creation module is deployed in the database system and is used to execute DDL statements using a two-phase commit method. In the first phase, the metadata is modified and marked as a specified intermediate state. In the second phase, the transaction is committed. It is also used to create a synchronously executed job before the transaction is committed in the second phase. The job contains a jobID as a unique identifier and the metadata of the objects involved in this DDL operation.

[0029] The job execution module is deployed in the database system and is used to send related distributed operations to the corresponding engine according to the DDL operation type;

[0030] The execution detection module is deployed in the database system to detect the status of the node executing the DDL operation. When it is detected that a node fails to execute or an abnormal downtime occurs, the rollback driver module is called. When it is detected that all nodes have successfully executed the DDL operation, the driver submission module is called;

[0031] The rollback driver module is deployed in the database system to drive all nodes to perform rollback operations when a node fails to execute or an abnormal downtime occurs;

[0032] The driver submission module is deployed in the database system to drive all nodes to perform the commit operation after all nodes have successfully executed the DDL operation, completing the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

[0033] Optionally, the execution creation module involved uses a two-phase commit method to execute DDL statements, specifically including:

[0034] For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation.

[0035] For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

[0036] Optionally, the job execution module involved sends the relevant distributed operation to the corresponding engine according to the DDL operation type. Before modifying the table structure, the engine needs to first record a WAL log entry containing the jobID.

[0037] Optionally, when the execution detection module involved detects that the engine execution of a certain node fails due to foreseeable reasons, the rollback driver module is called to trigger the rollback operation. All nodes participate in the rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback.

[0038] When the execution detection module detects that a node cannot return error information due to abnormal downtime, it calls the rollback driver module and triggers the rollback operation after the predetermined timeout period has expired. All nodes participate in this rollback process.

[0039] Further optionally, when the execution detection module involved detects that all nodes have successfully executed the DDL operation, it calls the driver submission module to drive the WAL modules of all nodes to perform the commit operation, complete the metadata modification of this DDL, and change the intermediate state of the object to the public state.

[0040] A method and system for ensuring the atomicity of DDL operations in a distributed database based on WAL of the present invention has the following beneficial effects compared with the prior art:

[0041] 1. The present invention ensures the orderly execution and reliability of operations and the atomicity and consistency of DDL operations by marking the status and creating synchronous execution tasks in the first stage, and sending distributed operations according to the operation type in the second stage; the intermediate state is used to manage the metadata change process to ensure that external users will not see incomplete or erroneous metadata status before the DDL operation is completed, avoiding the risk of providing inconsistent views to the outside during the DDL operation;

[0042] 2. When the DDL operation fails or the node goes down, the present invention can effectively drive all nodes to roll back, and accurately roll back through the WAL log and the unique jobID, thereby improving the recovery capability and data consistency of the database system in the event of a failure, reducing manual intervention, and improving the automation and stability of the database system;

[0043] 3. The present invention can send tasks to different engines according to the operation type, has the characteristics of adapting to multi-mode databases, allows different types of data storage and processing engines to work effectively together, and improves the overall performance of the database system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0045] Attached Figure 2 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0047] Embodiment 1:

[0048] Combined with Figure 1 This embodiment proposes a method for ensuring the atomicity of DDL operations in a distributed database based on WAL, which includes the following steps:

[0049] S1. The database system uses a two-phase commit method to execute DDL statements, including:

[0050] For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation.

[0051] For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

[0052] After the first phase is completed and before the transaction is committed in the second phase, the database system creates a synchronously executed job. The job contains a jobID as a unique identifier and metadata of the objects involved in this DDL operation.

[0053] S2. When executing a job, the database system sends the relevant distributed operations to the corresponding engine according to the DDL operation type, such as the TS Engine. Before modifying the table structure, the engine needs to first record the WAL log entry containing the jobID.

[0054] S3: When a node fails to execute or crashes during the job execution, all nodes are driven to perform rollback operations, including:

[0055] (a) When the engine execution of a node fails due to foreseeable reasons, the database system will trigger a rollback operation. All nodes participate in this rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback;

[0056] (b) For nodes that cannot return error information due to abnormal downtime, the database system triggers a rollback operation after a predetermined timeout period has passed, and all nodes participate in this rollback process.

[0057] S4. After all nodes successfully execute the DDL operation, the upper layer of the database system drives all nodes to perform the commit operation to complete the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

[0058] After the WAL modules of all nodes complete the commit operation, the upper layer of the database system will complete the metadata modification of this DDL and change the intermediate state of the object to the public state.

[0059] Embodiment 2:

[0060] Combined with Figure 2 This embodiment proposes a system for ensuring the atomicity of DDL operations in a distributed database based on WAL, which includes:

[0061] The execution creation module is deployed in the database system and is used to execute DDL statements using a two-phase commit method. In the first phase, the metadata is modified and marked as a specified intermediate state. In the second phase, the transaction is committed. It is also used to create a synchronously executed job before the transaction is committed in the second phase. The job contains a jobID as a unique identifier and the metadata of the objects involved in this DDL operation.

[0062] The job execution module is deployed in the database system and is used to send related distributed operations to the corresponding engine according to the DDL operation type, such as the TS Engine. Before modifying the table structure, the engine needs to first record the WAL log entry containing the jobID;

[0063] The execution detection module is deployed in the database system to detect the status of the node executing the DDL operation. When it is detected that a node fails to execute or an abnormal downtime occurs, the rollback driver module is called. When it is detected that all nodes have successfully executed the DDL operation, the driver submission module is called;

[0064] The rollback driver module is deployed in the database system to drive all nodes to perform rollback operations when a node fails to execute or an abnormal downtime occurs;

[0065] The driver submission module is deployed in the database system to drive all nodes to perform the commit operation after all nodes have successfully executed the DDL operation, completing the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

[0066] In this embodiment, the execution creation module involved uses a two-phase commit method to execute DDL statements, specifically including:

[0067] For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation.

[0068] For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

[0069] In this embodiment, when the execution detection module detects that the engine execution of a certain node fails due to foreseeable reasons, the rollback driver module is called to trigger the rollback operation. All nodes participate in the rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback.

[0070] When the execution detection module detects that a node cannot return error information due to abnormal downtime, it calls the rollback driver module and triggers the rollback operation after the predetermined timeout period has expired. All nodes participate in this rollback process.

[0071] In this embodiment, when the execution detection module detects that all nodes have successfully executed the DDL operation, it calls the driver submission module to drive the WAL modules of all nodes to perform the commit operation, complete the metadata modification of this DDL, and change the intermediate state of the object to the public state.

[0072] In summary, the method and system for ensuring the atomicity of DDL operations in a distributed database based on WAL of the present invention can ensure the orderly execution and reliability of operations, and the atomicity and consistency of DDL operations by marking the status and creating synchronous execution tasks in the first stage, and sending distributed operations according to the operation type in the second stage; the intermediate state is used to manage the metadata change process, ensuring that external users will not see incomplete or erroneous metadata status before the DDL operation is completed, avoiding the risk of providing inconsistent views to the outside during the DDL operation.

[0073] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method and system for ensuring the atomicity of DDL operations in a distributed database based on WAL, characterized in that: The steps include: S1. The database system executes DDL statements using a two-phase commit method. In the first phase, the metadata is modified and marked as a specified intermediate state. In the second phase, the transaction is committed. Before the transaction is committed in the second phase, the database system creates a synchronously executed job. The job contains a jobID as a unique identifier and the metadata of the objects involved in this DDL operation. S2. When executing a job, the database system sends the relevant distributed operations to the corresponding engine according to the DDL operation type; S3: When a node fails to execute or crashes during the job execution, all nodes are driven to perform rollback operations; S4. After all nodes successfully execute the DDL operation, the upper layer of the database system drives all nodes to perform the commit operation to complete the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

2. A method for ensuring the atomicity of DDL operations in a distributed database based on WAL according to claim 1, characterized in that: Execute step S1, the database system uses a two-phase commit method to execute the DDL statement, specifically including: For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation. For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

3. A method for ensuring the atomicity of DDL operations in a distributed database based on WAL according to claim 1, characterized in that: Execute step S2. Before modifying the table structure, the engine needs to first record a WAL log entry containing the jobID.

4. A method for ensuring the atomicity of DDL operations in a distributed database based on WAL according to claim 1, characterized in that: Execute step S3. When the engine execution of a node fails due to foreseeable reasons, the database system will trigger a rollback operation. All nodes participate in this rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback. For nodes that cannot return error information due to abnormal downtime, the database system triggers a rollback operation after a predetermined timeout period has passed, and all nodes participate in the rollback process.

5. A method for ensuring the atomicity of DDL operations in a distributed database based on WAL according to claim 4, characterized in that: Execute step S4. After the WAL modules of all nodes complete the commit operation, the upper layer of the database system will complete the metadata modification of this DDL and change the intermediate state of the object to the public state.

6. A system for ensuring the atomicity of DDL operations in a distributed database based on WAL, characterized in that: It includes: The execution creation module is deployed in the database system and is used to execute DDL statements using a two-phase commit method. In the first phase, the metadata is modified and marked as a specified intermediate state. In the second phase, the transaction is committed. It is also used to create a synchronously executed job before the transaction is committed in the second phase. The job contains a jobID as a unique identifier and the metadata of the objects involved in this DDL operation. The job execution module is deployed in the database system and is used to send related distributed operations to the corresponding engine according to the DDL operation type; The execution detection module is deployed in the database system to detect the status of the node executing the DDL operation. When it is detected that a node fails to execute or an abnormal downtime occurs, the rollback driver module is called. When it is detected that all nodes have successfully executed the DDL operation, the driver submission module is called; The rollback driver module is deployed in the database system to drive all nodes to perform rollback operations when a node fails to execute or an abnormal downtime occurs; The driver submission module is deployed in the database system to drive all nodes to perform the commit operation after all nodes have successfully executed the DDL operation, completing the final submission of the DDL operation. At this time, the metadata changes are visible to the outside world and take effect.

7. A system for ensuring the atomicity of distributed database DDL operations based on WAL according to claim 6, characterized in that: The execution creation module executes DDL statements using a two-phase commit method, specifically including: For CREATE / ALTER statements, the database system completes the metadata modification in the first phase of the DDL operation and marks the newly added or modified metadata with an intermediate state. The metadata in the intermediate state is not visible to the outside world, and the transaction is committed in the second phase of the DDL operation. For a DROP statement, the database system does not delete metadata in the first phase of the DDL operation. It only marks the metadata to be deleted as DROP and commits the transaction in the second phase of the DDL operation.

8. A system for ensuring the atomicity of DDL operations in a distributed database based on WAL according to claim 6, characterized in that: The job execution module sends the relevant distributed operation to the corresponding engine according to the DDL operation type. Before modifying the table structure, the engine needs to first record the WAL log entry containing the jobID.

9. A system for ensuring the atomicity of distributed database DDL operations based on WAL according to claim 6, characterized in that: When the execution detection module detects that the engine execution of a node fails due to foreseeable reasons, the rollback driver module is called to trigger the rollback operation. All nodes participate in the rollback process. During the rollback, the WAL module uses the jobID to locate the previous DDL operation to achieve accurate rollback. When the execution detection module detects that a node cannot return error information due to abnormal downtime, the rollback driving module is called to trigger a rollback operation after a predetermined timeout period has passed, and all nodes participate in the rollback process.

10. A system for ensuring the atomicity of distributed database DDL operations based on WAL according to claim 9, characterized in that: When the execution detection module detects that all nodes have successfully executed the DDL operation, it calls the driver submission module to drive the WAL modules of all nodes to perform the commit operation, complete the metadata modification of this DDL, and change the intermediate state of the object to the public state.

Images