A ddl operation framework and system

Abstract

The application provides a DDL operation framework and system, the framework comprises: a stub module used for saving an obtained DDL execution statement and data of the operation framework; and a round processing module used for interacting with the data of the stub module; wherein the round processing module comprises at least one sub-module, each sub-module is configured with a to-be-processed chain table, an execution success chain table and an execution failure chain table; and based on a statement type of the DDL execution statement, a reverse operation statement corresponding to the DDL execution statement is generated in the to-be-processed chain table. The application can be implemented in a plug-in, the original code file is not modified, there is no version upgrade problem, and whether to start a transaction can be flexibly selected for each round. All rounds are packaged into a stub in sequence for unified management. When an error is encountered, the stub automatically performs a rollback or a predefined reverse operation according to recorded state data.

Description

Technical Field

[0001] This invention relates to the field of database cluster technology, and in particular to a DDL operation framework and system. Background Technology

[0002] When transforming a single-machine database into a distributed database, DDL statements that originally ran in a single database need to run across the cluster. Due to network failures, system failures, or hardware failures, sometimes DDL statements cannot all be executed successfully across the cluster. In this case, to ensure cluster consistency, it is necessary to ensure that all operations are rolled back or reversed normally. When multiple DDL statements need to be executed across the cluster, errors in any statement need to be handled, making rollback or reversal operations extremely complex.

[0003] Data Definition Language (DDL) statements are used to describe real-world entities to be stored in a database. A database transaction is a sequence of database operations that access and potentially manipulate various data items; these operations are either all executed or none are executed, making it an indivisible unit of work. A transaction consists of all database operations performed between its start and end. Most DDL statements can run within a transaction, while a few, such as database creation statements in PostgreSQL, cannot. Two-phase commit (2PC) is a classic strongly consistent centralized replica control protocol. In this protocol, participating nodes are divided into two categories: a centralized coordinator node and N participant nodes. In the first phase, the coordinator asks all participants whether the transaction can be committed (participants vote), and all participants vote for the coordinator. In the second phase, the coordinator decides whether the transaction can be globally committed based on the votes of all participants and notifies all participants to execute the decision. In a two-phase commit process, participants cannot change their votes. The two-phase commit protocol allows for global commit only if all participants agree to commit the transaction. If even one participant votes to abort the transaction, the transaction must be abandoned.

[0004] Currently, the traditional approach involves significantly modifying the database kernel source code to support DDL cluster operations. This primarily involves: expanding the DDL syntax at the kernel level to distinguish between local and cluster statements; establishing communication links for each node during statement processing in the kernel and forwarding the statement to remote nodes for execution; obtaining the execution results from remote nodes and comprehensively determining whether the statement was executed successfully. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that, in the prior art, statement processing in distributed databases requires familiarity with database principles and a deep understanding of the source code details of the database, which has a high technical threshold. Moreover, when the database undergoes major version updates, it is difficult to keep up with the upgrade due to the large amount of code changes. In view of this, the present invention provides a DDL operation framework and system.

[0006] The technical solution adopted in this invention is a DDL operation framework, comprising:

[0007] The stub module is used to save the acquired DDL execution statements and operation framework data;

[0008] At least one round processing module is used to interact with the stub module, and the DDL execution statements in each round processing module are processed in the same batch;

[0009] The round processing module includes at least one sub-module, and each sub-module is configured with a pending processing list, a successful execution list, and a failed execution list.

[0010] Furthermore, when the DDL execution statement is stored in the pending list, a reverse operation statement corresponding to the DDL execution statement is generated in the pending list based on the statement type of the DDL execution statement; in response to the processing request of the DDL execution statement, the DDL execution statement in the pending list is executed, the successfully executed DDL execution statement is stored in the execution success list, and the failed DDL execution statement is stored in the execution failure list.

[0011] In one implementation, the stub module and its corresponding round processing module are on the same cluster node; when there are multiple DDL processing tasks to be implemented in the cluster, the different DDL processing tasks are assigned to different nodes in the cluster, and the stub module and its corresponding round processing module are created on the corresponding nodes.

[0012] In one implementation, the submodule is further configured as follows:

[0013] When the DDL execution statement is executed successfully, the corresponding DDL execution statement is deleted from the pending list and saved to the successful execution list.

[0014] When the execution of the DDL statement fails, the processing of the DDL statement is stopped, the corresponding DDL statement is deleted from the pending list, and it is saved to the execution failure list.

[0015] In one implementation, the round processing module is further configured to select whether to initiate a transaction;

[0016] When the round processing module starts a transaction and all the DDL execution statements are executed successfully, it is recorded as a success within the transaction;

[0017] When the round processing module has not started a transaction and all the DDL execution statements are executed successfully, it is recorded as a success outside the transaction;

[0018] When the round processing module starts a transaction and there is a failed DDL execution statement, it is recorded as a failure within the transaction;

[0019] When the round processing module has not started a transaction and there is a failed DDL execution statement, it is recorded as an out-of-transaction failure.

[0020] In one embodiment, the round processing module is further configured to: further process the execution status of the current round in the round processing module, including:

[0021] When a transaction succeeds, send a commit preparation message.

[0022] If an out-of-transaction success occurs, no further processing is performed in this round;

[0023] If a transaction fails, no further processing is performed in this round; the reverse operation statements corresponding to the DDL processing statements in the successful execution list of previous rounds are executed in reverse order.

[0024] When an out-of-transaction failure occurs, the reverse operation statement corresponding to the DDL processing statement in the failure chain is executed first in this round; then the reverse operation statement corresponding to the DDL processing statement in the success chain is executed; finally, the reverse operation statement corresponding to the DDL processing statement in the success chain from previous rounds is executed in reverse order.

[0025] In one implementation, the round processing module saves the data in a doubly linked list in the form of a connection structure according to the order in which each sub-module is opened, with strings as labels and node names as key values.

[0026] Another aspect of the present invention provides a DDL operating system, the system comprising the DDL operation framework as described in any of the preceding claims.

[0027] Another aspect of the present invention provides an electronic device comprising the DDL operating system described above.

[0028] By adopting the above technical solution, the present invention has at least the following advantages:

[0029] This invention allows for implementation within a plugin without modifying existing code files, eliminating version upgrade issues. When executing cluster-wide DDL statements, the framework interface provided by this invention allows for the division of statements into several groups as needed. Each group of statements is packaged into a round, and each round selects one or more cluster nodes to participate in its execution. Furthermore, each round can flexibly choose whether to initiate transactions. All rounds are sequentially packaged into a stub for unified management. When an error occurs, the stub automatically performs a rollback or a predefined reverse operation based on the recorded status data. Compared to traditional solutions, this avoids source code modification and standardizes the required operations and processes through a framework interface, reducing knowledge requirements and coding difficulty, and facilitating the formation of a development team at a lower cost. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall process of cluster operation according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the composition structure of the DDL operation framework according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the actual application process of the DDL operation framework according to an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0034] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.

[0035] In the accompanying drawings, the thickness, size, and shape of the objects have been slightly exaggerated for ease of illustration. The drawings are for illustrative purposes only and are not drawn to scale.

[0036] It should also be understood that the terms "comprising," "including," "having," "containing," and / or "comprising," when used in this specification, indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire listed feature, not individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to an example or illustration.

[0037] As used herein, the terms “basically,” “approximately,” and similar terms are used as terms of approximation rather than terms of degree, and are intended to describe inherent biases in measured or calculated values ​​that will be recognized by those skilled in the art.

[0038] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms (e.g., those defined in common dictionaries) shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and shall not be interpreted in an idealized or overly formal sense unless expressly so specified herein.

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily to be strictly followed according to the step numbers; the execution order of the steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0041] The first embodiment of the present invention, with reference to Figure 1 A DDL operation framework, including

[0042] The stub module is used to save the acquired DDL execution statements and operation framework data;

[0043] At least one round processing module is used to interact with the stub module, and the DDL execution statements in each round processing module are processed in the same batch;

[0044] The round processing module includes at least one sub-module, and each sub-module is configured with a pending processing list, a successful execution list, and a failed execution list.

[0045] Furthermore, when the DDL execution statement is stored in the pending list, a reverse operation statement corresponding to the DDL execution statement is generated in the pending list based on the statement type of the DDL execution statement; in response to the processing request of the DDL execution statement, the DDL execution statement in the pending list is executed, the successfully executed DDL execution statement is stored in the execution success list, and the failed DDL execution statement is stored in the execution failure list.

[0046] The functions of each node module will be explained in detail below.

[0047] First, it's important to note that the cluster DDL operation framework further encapsulates and extends the two-phase commit protocol, ensuring that all nodes can easily write code when performing DDL statement operations. The overall cluster operation flow is as follows: Figure 2 Obviously, users only need to focus on input.

[0048] Figure 1 In this context, DDE_Stub serves as the stub module, which is used to store all data from the entire DDL cluster operation.

[0049] Furthermore, for each additional round, a DDE_Round is added to the bidirectional list dlRound in the stub module configuration, which is the round processing module mentioned above.

[0050] Furthermore, the links opened in a round are stored in a doubly linked list dlNodeJoin in the form of a connection structure according to the order in which they were opened. Different connection structures are each a sub-module in the processing module of that round. The sub-modules are labeled with the string node_tag and the node name nod_name as the key value for easy searching.

[0051] Furthermore, for a submodule, there are three doubly linked lists ddlsWait, ddlsOK, and ddlsFailed, which are used for statements to be executed, statements that were executed successfully, and statements that were executed unsuccessfully, respectively, i.e., the pending list, the successful execution list, and the failed execution list mentioned above.

[0052] Specifically, each linked list stores a structure called DDE_SqlObj, which contains two character arrays, used to store the original DDL statement and the reverse operation statement, respectively.

[0053] In this embodiment, the stub module and its corresponding round processing module are on the same cluster node; when there are multiple DDL processing tasks to be implemented in the cluster, the different DDL processing tasks are assigned to different nodes in the cluster, and the stub module and its corresponding round processing module are created on the corresponding nodes.

[0054] In one implementation, the submodule can be further configured as follows:

[0055] When a DDL execution statement is executed successfully, the corresponding DDL execution statement is deleted from the pending list and saved to the successful execution list.

[0056] When a DDL execution statement fails, processing of the DDL execution statement is stopped, the corresponding DDL execution statement is deleted from the pending list, and it is saved to the execution failure list.

[0057] In this embodiment, the round processing module can independently choose whether to start a transaction, and it can be specifically used for:

[0058] When the round processing module starts a transaction and all DDL execution statements are executed successfully, it is recorded as a success within the transaction;

[0059] When the round processing module has not started a transaction and all DDL execution statements are executed successfully, it is recorded as a success outside the transaction;

[0060] When the round processing module starts a transaction and there are DDL execution statements that fail to execute, it is recorded as a failure within the transaction;

[0061] When the round processing module has not started a transaction and there is a failed DDL execution statement, it is recorded as an out-of-transaction failure.

[0062] Furthermore, the execution status within the current round of the round processing module can be further processed, including:

[0063] When a transaction succeeds, the commit preparation information is sent in this round;

[0064] If an out-of-transaction success occurs, no further processing is performed in this round;

[0065] If a transaction fails, no further processing is performed in this round; the reverse operation statements corresponding to the DDL processing statements in the successful execution list of previous rounds are executed in reverse order.

[0066] When an out-of-transaction failure occurs, the reverse operation statement corresponding to the DDL processing statement in the failure list is executed first in this round; then the reverse operation statement corresponding to the DDL processing statement in the success list is executed; finally, the reverse operation statement corresponding to the DDL processing statement in the success list as described in previous rounds is executed in reverse order.

[0067] By adopting the above technical solution, this embodiment has at least the following advantages:

[0068] This embodiment allows for implementation within the plugin without modifying existing code files, eliminating version upgrade issues. When executing cluster-wide DDL statements, the framework interface provided by this invention allows for the division of statements into several groups as needed. Each group of statements is packaged into a round, and each round selects one or more cluster nodes to participate in its execution. Furthermore, each round can flexibly choose whether to initiate transactions. All rounds are sequentially packaged into a stub for unified management. When an error occurs, the stub automatically performs a rollback or a predefined reverse operation based on the recorded status data. Compared to traditional solutions, this avoids source code modification and standardizes the required operations and processes through a framework interface, reducing knowledge requirements and coding difficulty, and facilitating the formation of a development team at a lower cost.

[0069] The second embodiment of the present invention, as follows: Figure 3 As shown, this embodiment corresponds to the first embodiment and provides a practical application method of the DDL operation framework based on the framework provided in the first embodiment.

[0070] Step 1: Create a stub using DDE_MakeStub to store all the data;

[0071] Step 2: Create a round using DDE_AddRound. Statements to be executed in the same batch will be placed in the same round. If this round needs to be executed within a transaction, a unique and random string is created using DDE_MakeGidViaCurNode. This string is used for two-phase commit.

[0072] Step 3: Open connections to all nodes using DDE_BeginAllXNinRound and save the connections and their corresponding nodes for later use. If some statements only need to operate on a specific node, then open a connection to that specific node using DDE_Begin.

[0073] Step 4: After analyzing multiple DDL statements using DDE_ExecDList, their statement types, operands, and original statements are saved into a specific structure. This structure is then placed in the execution list and executed sequentially. Before executing a statement, a reverse operation statement is automatically constructed based on its statement type; for example, a table creation statement is concatenated to a table deletion statement. Successfully executed statements are added to the success list and removed from the execution list. Failed statements are added to the failure list, and execution of the lists is stopped.

[0074] Step 5: Use DDE_SaveTemporarily to determine if all statements to be executed have been executed successfully: If executed within a transaction and successfully, send a prepare transaction command and record the status as transaction successful; if executed within a transaction and fails, close the connection and record the status as transaction failed; if executed outside a transaction and successfully, do nothing and record success outside the transaction; if executed outside a transaction and fails, record failure outside the transaction.

[0075] Step 6: Perform the final determination using DDE_EndWithDecide:

[0076] (1) Judgment in this round: If the transaction is successful, send commit prepared; if the transaction fails, do nothing; if the transaction is successful outside the transaction, do nothing; if the transaction fails outside the transaction, first execute the reverse operation statement in the failure list, and then execute the reverse operation statement in the success list.

[0077] (2) Previous rounds: Reverse the operation statements in the list of successful reverse executions.

[0078] According to a third embodiment of the present invention, a DDL processing system includes a DDL processing framework as provided in the first embodiment, which can be used to execute the DDL processing method as provided in the second embodiment.

[0079] Fourth embodiment of the present invention, an electronic device, such as Figure 4 As shown, it can be understood as a physical device, including the DDL processing system provided by the third embodiment.

[0080] In summary, compared with the prior art, the present invention has at least the following advantages:

[0081] 1) The original code files have not been modified, so there is no issue with version upgrades;

[0082] 2) Divide multiple DDL statements into multiple rounds for execution, and each round can flexibly choose whether to start a transaction;

[0083] 3) Automatically rollback or perform predefined reverse operations when an error is encountered;

[0084] 4) It reduced the knowledge requirements and coding difficulty.

[0085] Through the description of specific embodiments, a more in-depth and specific understanding should be gained of the technical means and effects adopted by the present invention to achieve the intended purpose. However, the accompanying drawings are only provided for reference and illustration and are not intended to limit the present invention.

Claims

1. A DDL operating system, characterized in that, include: The stub module is used to save the acquired DDL execution statements and operation framework data; At least one round processing module is used to interact with the stub module, and the DDL execution statements in each round processing module are processed in the same batch; The round processing module includes at least one sub-module, and each sub-module is configured with a pending processing list, a successful execution list, and a failed execution list. Furthermore, when the DDL execution statement is stored in the pending list, based on the statement type of the DDL execution statement, a reverse operation statement corresponding to the DDL execution statement is generated in the pending list; in response to the processing request of the DDL execution statement, the DDL execution statement in the pending list is executed, the successfully executed DDL execution statement is stored in the successful execution list, and the failed DDL execution statement is stored in the failed execution list. The stub module and its corresponding round processing module are on the same cluster node; when there are multiple DDL processing tasks to be implemented in the cluster, the different DDL processing tasks are assigned to different nodes of the cluster, and the stub module and its corresponding round processing module are created on the corresponding nodes. The submodule is further configured as follows: When the DDL execution statement is executed successfully, the corresponding DDL execution statement is deleted from the pending list and saved to the successful execution list. When the execution of the DDL statement fails, the processing of the DDL statement is stopped, the corresponding DDL statement is deleted from the pending list, and it is saved to the execution failure list. The round processing module is further used to select whether to start a transaction; When the round processing module starts a transaction and all the DDL execution statements are executed successfully, it is recorded as a success within the transaction; When the round processing module has not started a transaction and all the DDL execution statements are executed successfully, it is recorded as a success outside the transaction; When the round processing module starts a transaction and there is a failed DDL execution statement, it is recorded as a failure within the transaction; When the round processing module has not started a transaction and there is a failed DDL execution statement, it is recorded as an out-of-transaction failure.

2. The DDL operating system according to claim 1, characterized in that, The round processing module is further configured to: further process the execution status of the current round in the round processing module, including: When a transaction succeeds, the commit preparation information is sent in this round; If an out-of-transaction success occurs, no further processing is performed in this round; If a transaction fails, no further processing is performed in this round; the reverse operation statements corresponding to the DDL processing statements in the successful execution list of previous rounds are executed in reverse order. When an out-of-transaction failure occurs, the reverse operation statement corresponding to the DDL processing statement in the failure chain is executed first in this round; then the reverse operation statement corresponding to the DDL processing statement in the success chain is executed; finally, the reverse operation statement corresponding to the DDL processing statement in the success chain from previous rounds is executed in reverse order.

3. The DDL operating system according to claim 1, characterized in that, In the round processing module, the sub-modules are opened in the order they are connected and stored in a doubly linked list in the form of a connection structure, with strings as labels and node names as key values.

4. An electronic device, characterized in that, The electronic device includes the DDL operating system as described in any one of claims 1-3.

Images