A Metadata Change Capture Method

Abstract

The present invention relates to the technical field of database data capture, and specifically provides a metadata change capture method, which has the following steps: S1, executing CDC statements in a distributed database with a metadata capture function; S2, a user executes a DDL operation, and the database service syntax parses SQL to generate a syntax tree, and then generates an execution plan through the syntax tree; S3, the database sends the execution plan to each execution node for execution, and after the plan is executed, the metadata change record is updated to the system table, and the monitor pushes these metadata change records to the changefeed; S4, judging whether the changefeed sending mode is the DDL sending mode, if yes, directly sending the DDL; if not, waiting for the sending mode to switch to the DDL sending mode before sending. Compared with the prior art, the present invention does not need to change the decentralized structure of the database, does not rely on the timing service of the central node, and ensures the transaction semantics of the DDL is captured through a series of judgments, and does not rely on the central node database to still maintain good scalability.

Description

Technical Field

[0001] The present invention relates to the technical field of database data capture, and in particular provides a metadata change capture method. Background Art

[0002] In large-scale distributed database systems, capturing metadata changes faces great challenges. We hope to be able to scale infinitely with the cluster size while ensuring strong transaction semantics. Currently, the CDC function of known databases mainly adopts the following two strategies for capturing metadata changes:

[0003] (1) Extract log files through external programs and reorganize transactions externally

[0004] This solution establishes an external CDC cluster dedicated to extracting, analyzing and sending data, extracts database logs and reorders the content to ensure the transaction semantics of the sent data.

[0005] (2) Only supports capturing data changes, not metadata changes

[0006] This solution assumes that metadata changes are rare operations during data synchronization, and the DBA can manually modify the structure of the target database to ensure the success of data migration.

[0007] Solution (1) requires recording additional logs, taking up additional system process resources, increasing IO pressure, and mostly relying on central nodes to ensure high availability of external programs. Solution (2) does not support the capture of metadata changes and relies on manual transformation of the target database structure, which brings unknown risks to data migration synchronization. Summary of the invention

[0008] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a highly practical metadata change capture method.

[0009] The technical solution adopted by the present invention to solve its technical problem is:

[0010] A metadata change capture method has the following steps:

[0011] S1, execute CDC statements in distributed databases with metadata capture function;

[0012] S2. The user executes the DDL operation, and the database service parses the SQL to generate a syntax tree, and then generates an execution plan based on the syntax tree.

[0013] S3. The database sends the execution plan to each execution node for execution. After the plan is executed, the metadata change records are updated to the system table, and the monitor pushes these metadata change records to the changefeed;

[0014] S4. Determine whether the changefeed sending mode is the DDL sending mode. If yes, send the DDL directly; if not, wait until the sending mode is switched to the DDL sending mode before sending.

[0015] Furthermore, in step S1, the specific operations of creating a CDC include:

[0016] S101. Create a CDC statement;

[0017] S102. After CDC starts capturing metadata changes, it will start monitoring the system table that records table structure changes within the database.

[0018] Furthermore, in step S2, after the execution plan is generated, the CDC registration information of the operation object is first queried to determine whether the CDC has been created.

[0019] Furthermore, in step S2, the step of determining whether to create a CDC is as follows:

[0020] a) If CDC is not created, a normal execution plan is generated;

[0021] b) If it is found that the CDC has been created, a CDC processor is generated according to the CDC registration information, and the execution plan and the CDC processor are encapsulated as a derived object of the execution plan.

[0022] Furthermore, in step S3, it further includes:

[0023] S301. If a CDC processor exists and metadata change capture is enabled, metadata change records are pushed to the changefeed:

[0024] S302: If there is no CDC processor or the metadata change capture is not enabled, the transaction is directly committed.

[0025] Furthermore, in step S4, it further includes:

[0026] S401. If the current changefeed sending mode is the DDL sending mode, the DDL is sent to the specified location, and the DML in the temporary file is read, and then the changefeed state is updated to the DML sending mode.

[0027] S402: If the current changefeed sending mode is other modes, it is necessary to block the DDL sending process and wait for the sending mode to be updated to the DDL sending mode by other processes.

[0028] Furthermore, the following conditions must be met for changefeed to send DML:

[0029] (1) The current sending mode is DML sending mode;

[0030] (2) lastDDLSendTime is less than ModificationTime;

[0031] If the above conditions are not met, the DML will not be sent directly but recorded in a temporary file. The DML content in the temporary file will be read after the DDL is sent.

[0032] Furthermore, the changefeed mode switching must meet the following conditions:

[0033] (1) ChangeFeedSyncDML is switched to ChangeFeedExecDDL

[0034] a) The blockTime corresponding to the DDL is not empty, which means that there is currently a blocked DDL waiting to be sent;

[0035] b) lastDMLTransactionCommitTime is less than or equal to sfTime, which means that the last DML transaction commit time is less than the resolution time of each range;

[0036] (2) ChangeFeedExecDDL is converted to ChangeFeedWaitToExecDDL; metadata changes are captured but not sent successfully;

[0037] (3) ChangeFeedWaitToExecDDL is converted to ChangeFeedSyncDML;

[0038] Metadata changes have been sent successfully.

[0039] Furthermore, a CDC processor with metadata change monitoring is started asynchronously to monitor metadata changes and send data to the data pool.

[0040] Furthermore, the CDC processor with metadata change monitoring is started asynchronously to monitor metadata changes and send data to the data pool. The specific operations of this process are as follows:

[0041] a) When starting monitoring of the user table, start a special monitoring program to monitor the system table that records changes in the table structure;

[0042] b) Parse and reorganize the captured data and filter it based on the unique identifier of the table;

[0043] c) Connect to the external data pool and output the table metadata changes to the external data pool.

[0044] Compared with the prior art, the metadata change capture method of the present invention has the following outstanding beneficial effects:

[0045] (1) The present invention can well realize the capture of table structure changes without changing the decentralized structure of the database and without relying on the timing service of the central node. Instead, the sending mode is switched through the timestamps carried by DDL and DML. The transaction semantics of DDL are captured through a series of judgments. The present invention does not rely on the central node database and still maintains good scalability.

[0046] (2) The present invention does not rely on an additional log system, reduces the storage space occupied, reduces the computing and IO resource occupation, reduces the network load, improves the change data capture efficiency and real-time performance, and improves system performance.

[0047] (3) Compared with the CDC function that does not support capturing metadata changes, the present invention will not cause migration and synchronization task failures due to metadata changes during data migration and synchronization, and does not require the DBA to manually migrate the DDL, thereby providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Attached Figure 1 It is a schematic diagram of the process in step S1 of the metadata change capture method;

[0050] Attached Figure 2 is a flow chart of steps S2-S4 in a metadata change capture method;

[0051] Attached Figure 3 It is a flow chart of the CDC processor processing DDL in the metadata change capture method;

[0052] Attached Figure 4 The diagram is a flow chart of a CDC processor processing DML in a metadata change capture method. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] A best embodiment is given below:

[0055] like Figure 1-4 As shown, a metadata change capture method in this embodiment has the following steps:

[0056] S1, execute CDC statements in distributed databases with metadata capture function;

[0057] S2. The user executes the DDL operation, and the database service parses the SQL to generate a syntax tree, and then generates an execution plan based on the syntax tree.

[0058] S3. The database sends the execution plan to each execution node for execution. After the plan is executed, the metadata change records are updated to the system table, and the monitor pushes these metadata change records to the changefeed;

[0059] S4. Determine whether the changefeed sending mode is the DDL sending mode. If yes, send the DDL directly. If not, wait until the sending mode is switched to the DDL sending mode before sending.

[0060] In step S1, the specific operations of creating a CDC include:

[0061] S101. Create a CDC statement;

[0062] S102. After CDC starts capturing metadata changes, it will start monitoring the system table that records table structure changes within the database.

[0063] In step S2, after the execution plan is generated, the CDC registration information of the operation object is first queried to determine whether the CDC has been created;

[0064] a) If CDC is not created, a normal execution plan is generated;

[0065] b) If it is found that the CDC has been created, a CDC processor is generated according to the CDC registration information, and the execution plan and the CDC processor are encapsulated as a derived object of the execution plan.

[0066] In step S3, further comprising:

[0067] S301. If a CDC processor exists and metadata change capture is enabled, metadata change records are pushed to the changefeed:

[0068] S302: If there is no CDC processor or the metadata change capture is not enabled, the transaction is directly committed.

[0069] In step S4, further comprising:

[0070] S401. If the current changefeed sending mode is the DDL sending mode, the DDL is sent to the specified location, and the DML in the temporary file is read, and then the changefeed state is updated to the DML sending mode.

[0071] S402: If the current changefeed sending mode is other modes, it is necessary to block the DDL sending process and wait for the sending mode to be updated to the DDL sending mode by other processes.

[0072] Among them, changefeed sending DML needs to meet the following conditions:

[0073] (1) The current sending mode is DML sending mode;

[0074] (2) lastDDLSendTime is less than ModificationTime;

[0075] If the above conditions are not met, the DML will not be sent directly but recorded in a temporary file. The DML content in the temporary file will be read after the DDL is sent.

[0076] The following conditions must be met for the changefeed mode switch:

[0077] (1) ChangeFeedSyncDML is switched to ChangeFeedExecDDL

[0078] a) The blockTime corresponding to the DDL is not empty, which means that there is currently a blocked DDL waiting to be sent;

[0079] b) lastDMLTransactionCommitTime is less than or equal to sfTime, which means that the last DML transaction commit time is less than the resolution time of each range;

[0080] (2) ChangeFeedExecDDL is converted to ChangeFeedWaitToExecDDL;

[0081] Metadata changes were captured but not yet successfully sent;

[0082] (3) ChangeFeedWaitToExecDDL is converted to ChangeFeedSyncDML;

[0083] Metadata changes have been sent successfully.

[0084] The specific format of DDL is:

[0085] {

[0086] "Schema": <name of the schema where the table is located>,

[0087] "TableName": ,

[0088] "TableID": ,

[0089] "BeforeTableName": <rename the table, the name of the table before renaming>,

[0090] "Column": <column descriptor, when creating a table, it is all the columns of the table. Adding and deleting columns is corresponding columns>,

[0091] …

[0092] Different operations correspond to different changes

[0093] …

[0094] "Operate": <operation type>,

[0095] "Ts": <LOCAL TIME of the assembled DDL statement>,

[0096] }

[0097] Asynchronously start the CDC processor with metadata change monitoring, monitor metadata changes, and send data to the data pool. The specific operations of this process are:

[0098] a) When starting monitoring of the user table, start a special monitoring program to monitor the system table that records changes in the table structure;

[0099] b) Parse and reorganize the captured data and filter it based on the unique identifier of the table;

[0100] c) Connect to the external data pool and output the table metadata changes to the external data pool.

[0101] The specific format of capturing DDL and reorganizing it and outputting it to the data pool is:

[0102] {

[0103] "Schema": <name of the schema where the table is located>,

[0104] "TableName": ,

[0105] "TableID": ,

[0106] "BeforeTableName": <rename the table, the name of the table before renaming>,

[0107] "Column": <column descriptor, when creating a table, it is all the columns of the table. Adding and deleting columns is corresponding columns>,

[0108] …

[0109] Different operations correspond to different descriptors

[0110] …

[0111] "Operate": <operation type>,

[0112] "Ts": <LOCAL TIME of the assembled DDL statement>,

[0113] }

[0114] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any appropriate changes or substitutions made by any ordinary technician in the technical field that conform to the claims of a metadata change capture method of the present invention shall fall within the patent protection scope of the present invention.

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A metadata change capture method, characterized in that: The following steps are involved: S1. Execute CDC statements in distributed databases with metadata capture capabilities; The specific operations for creating a CDC include: S101. Create a CDC statement; S102. After CDC starts capturing metadata changes, it will start monitoring the system table that records table structure changes within the database. S2. The user executes the DDL operation, and the database service parses the SQL to generate a syntax tree, and then generates an execution plan based on the syntax tree. After the execution plan is generated, the CDC registration information of the operation object is queried first to determine whether the CDC has been created; The steps to determine whether to create a CDC are: a) If CDC is not created, a normal execution plan is generated; b) If it is found that the CDC has been created, a CDC processor is generated according to the CDC registration information, and the execution plan and the CDC processor are encapsulated as a derived object of the execution plan; S3. The database sends the execution plan to each execution node for execution. After the plan is executed, the metadata change records are updated to the system table, and the monitor pushes these metadata change records to the changefeed; Further including: S301. If a CDC processor exists and metadata change capture is enabled, metadata change records are pushed to the changefeed: S302: If there is no CDC processor or metadata change capture is not enabled, the transaction is directly committed. S4. Determine whether the changefeed sending mode is the DDL sending mode. If yes, send the DDL directly; if not, wait until the sending mode is switched to the DDL sending mode before sending; Further including: S401, if the current changefeed sending mode is the DDL sending mode, send the DDL to the specified location, read the DML in the temporary file, and then update the changefeed status to the DML sending mode; S402: If the current changefeed sending mode is other modes, it is necessary to block the DDL sending process and wait for the sending mode to be updated to the DDL sending mode by other processes; Changefeed sends DML only if the following conditions are met: (1) The current sending mode is DML sending mode; (2) lastDDLSendTime is less than ModificationTime; If the above conditions are not met, the DML will not be sent directly but recorded in a temporary file. The DML content in the temporary file will be read after the DDL is sent. The following conditions must be met for the changefeed mode switch: (1) ChangeFeedSyncDML is switched to ChangeFeedExecDDL a) The blockTime corresponding to the DDL is not empty, which means that there is currently a blocked DDL waiting to be sent; b) lastDMLTransactionCommitTime is less than or equal to sfTime, which means that the last DML transaction commit time is less than the resolved time of each range; (2) ChangeFeedExecDDL is converted to ChangeFeedWaitToExecDDL; Metadata changes were captured but not yet successfully sent; (3) ChangeFeedWaitToExecDDL is converted to ChangeFeedSyncDML; Metadata changes have been sent successfully; Asynchronously start the CDC processor with metadata change monitoring, monitor metadata changes, and send data to the data pool. The specific operations of this process are: a) When starting monitoring of the user table, start a special monitoring program to monitor the system table that records changes in the table structure; b) Parse and reorganize the captured data and filter it based on the unique identifier of the table; c) Connect to the external data pool and output the table metadata changes to the external data pool.

Images