Incremental data synchronization resuming method and system

Abstract

The application discloses a kind of incremental data synchronization breakpoint continuation methods and systems, comprising, initialization incremental synchronization task, by CDC engine to source library is parsed, obtains the incremental change of source library;Judgment incremental change is DDL event or DML event, if incremental change is DDL event, then parse executed DDL statement, and update table metadata and table meta history data;If incremental change is DML event, then based on table meta history data obtains table metadata, according to table metadata resolution generates corresponding SQL statement;Connect target library, target end executes the SQL statement generated by parsing, after successful execution, update synchronization point and continue to parse incremental change from new point.The application realizes that incremental data synchronization in interruption period, breakpoint continuation in the case where source library exists DDL, greatly reduce the cost of synchronization fault handling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of databases, and particularly relates to an incremental data synchronization breakpoint resuming method and system. BACKGROUND

[0002] Incremental data synchronization is an important technology for data updating and replication, aiming to transmit changed data from a source to one or more targets to maintain the consistency and integrity of data. This technology is widely used in Internet enterprises and is an indispensable technology in scenarios such as database disaster recovery, database multi-active, real-time data analysis, and application cloud.

[0003] Common incremental data synchronization generally has two implementation methods, T+1 synchronization scheme and real-time synchronization scheme.

[0004] Among them, T+1 synchronization is based on a full synchronization, and then the incremental data generated in a fixed period is synchronized to the target end every fixed period. This synchronization method has a one-period delay of target library data compared with source library, and is suitable for scenarios with low real-time requirements.

[0005] Real-time synchronization is based on a full synchronization, and through real-time analysis of the incremental operations of the source library and application to the target end, the incremental data of the source library is synchronized to the target library in real time, which can ensure the real-time consistency or small difference of the data between the target end and the source end. However, incremental data synchronization is a long-term running process, and it is inevitable that the incremental synchronization process will be interrupted due to network failure, node failure and other problems. If there is a DDL operation performed on the source library during the interruption, the table structure is changed, and the traditional data synchronization method directly restarts the synchronization task from the breakpoint. Because the table structure corresponding to the breakpoint is different from the current actual table structure of the source library, when parsing the DML event, it may fail due to mismatch of the table structure, so it is difficult to continue the synchronization from the breakpoint, and a full synchronization needs to be performed again to continue the synchronization. If the source library has a large amount of data, the processing cost is high. SUMMARY

[0006] In view of the problem that after the interruption of incremental data synchronization, the source library executes a DDL event, and cannot resume from the breakpoint, the present application provides an incremental data synchronization breakpoint resuming method and system.

[0007] To achieve the above purpose, the present application provides the following scheme: an incremental data synchronization breakpoint resuming method, comprising:

[0008] Initializing an incremental synchronization task, parsing the source library through a CDC engine to obtain the incremental changes of the source library;

[0009] judging whether the incremental change is a DDL event or a DML event, if the incremental change is a DDL event, parsing an executed DDL statement and updating table metadata and table history data;

[0010] if the incremental change is a DML event, obtaining table metadata based on the table history data, and generating a corresponding SQL statement according to the table metadata;

[0011] connecting a target library, executing the generated SQL statement at a target end, updating a synchronization point after successful execution, and continuing to parse the incremental change from the new point.

[0012] Preferably, the process of initializing the incremental synchronization task includes an initially created incremental synchronization task and an incremental synchronization task restarted after interruption;

[0013] if it is the initially created incremental synchronization task, the initialization operation includes recording current point information and table metadata information of the source library in metadata records, and synchronizing inventory data of the table to be synchronized to the target end;

[0014] if it is the incremental synchronization task restarted after interruption, the initialization operation includes reading the synchronization point from the metadata records and executing incremental data processing from the synchronization point.

[0015] Preferably, the process of parsing the source library by the CDC engine to obtain the incremental change of the source library includes,

[0016] parsing the incremental log of the source library by the CDC engine to output a standard format event; wherein the event content of the standard format event includes point information, a change type, a table name involved in the change, field before and after the change, and executed SQL information; and the change type includes a DDL event and a DML event.

[0017] Preferably, if the incremental change is a DDL event, the process of parsing an executed DDL statement and updating table metadata and table history data includes,

[0018] if the incremental change is a DDL event, parsing an executed DDL statement, obtaining the type of the change, the fields involved, the indexes, and the increase and decrease of the constraints, then applying the change to the corresponding table metadata, and updating the table metadata; wherein the type of the change includes create, alter, and drop; and the fields involved include addition, deletion, and renaming of the fields;

[0019] Meanwhile, the updated table metadata is recorded into table metadata history data, different table metadata versions within a time window are recorded based on the table metadata history data, and the two latest table metadata versions are kept if no DDL change occurs in a time window.

[0020] Preferably, if the incremental change is a DML event, the process of obtaining table metadata based on the table metadata history data comprises,

[0021] If the incremental change is a DML event, the corresponding version of table metadata is queried in the table metadata history data according to the table name and the site information, is obtained from the cache first, is obtained from the external storage if not obtained from the cache, and the task enters an exception if not obtained from the external storage. The obtained table metadata is parsed to generate a corresponding SQL statement.

[0022] Preferably, when the number of consecutive times of entering an exception after not obtaining from the external storage reaches a preset number or an exception occurs in the process of executing the SQL statement generated by parsing and the number of consecutive times of entering an exception reaches a preset number, the process further comprises,

[0023] The synchronization task is restarted at a self-defined fixed time interval, and the synchronization continues from the last synchronization success site. If the number of consecutive times of entering an exception within a self-defined specified time period exceeds a preset threshold number, the task is stopped and the task is not restarted.

[0024] Preferably, the process of generating a corresponding SQL statement according to the table metadata comprises,

[0025] According to the corresponding version of table metadata history data, a corresponding SQL execution statement is generated for a DML event.

[0026] Preferably, the process of executing the SQL statement generated by parsing comprises,

[0027] If the DDL event is executed successfully, the synchronization site is updated and the incremental change is parsed from a new site.

[0028] If the DDL event fails, the corresponding table metadata in the table metadata history data is deleted.

[0029] The application also provides an incremental data synchronization breakpoint resume transmission system, comprising:

[0030] The Select module is used to obtain a specified synchronization point from the metadata record, to obtain and parse incremental data changes of the source library in real time from the synchronization point by using the CDC engine, to convert the incremental data changes into corresponding DML events or DDL events, and to pass the DML events and DDL events to the Load module; and the Select module is also used to receive a synchronization result returned by the Load module and to update the metadata record.

[0031] The Load module is connected with the Select module and is used to receive the DML events and DDL events passed by the Select module, to execute the DML events and DDL events in the target library, and to return an execution result to the Select module, wherein an ACK is returned if the execution is successful, and a ROLLBACK is returned if the execution fails.

[0032] The metadata record module is connected with the Select module and is used to record the point information SYN_POS and the table meta history data of the table.

[0033] Preferably, the point information is progress information of the current synchronization to the target end.

[0034] The SYN_POS is a position identifier of a certain event in a log in the database, and the SYN_POS is unique and monotonically increasing.

[0035] The table meta history data is used to record the metadata version history of the table and is divided into a cache and a persistent external storage.

[0036] The recording mode is as follows: scheme_name.table_name:(SYN_POS_1:Table_meta_1, timestamp:t1), (SYN_POS_2:Table_meta2, timestamp:t2), …, (SYN_POS_n:Table_meta_1, timestamp:tn).

[0037] In the formula, Table_meta_n represents the metadata of the table scheme_name.table_name at the point SYN_POS_n, including but not limited to field information, primary key information, index information and constraint information of the table, and tn represents a time stamp of a time corresponding to the point SYN_POS_n.

[0038] Compared with the prior art, the present application has the following advantages and technical effects:

[0039] The application records the changed position after the DDL change and the corresponding table metadata history version during the increment process, and after the fault restart, the corresponding version of the table metadata is searched according to the position information, so that the increment data synchronization during the interruption can realize the breakpoint continuation, even if the source library has DDL change, the processing cost of the synchronization fault is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for a purpose of explanations. The illustrative embodiments of the present application, as well as the explanations for the illustrative embodiments, are used to explain the present application and do not constitute improper limitations to the present application. In the drawings:

[0041] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure;

[0042] Figure 2 The system structure diagram of the embodiment of the present application is shown in the figure;

[0043] Figure 3 The table metadata version corresponding relationship diagram of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0044] It should be noted that the embodiments and the features in the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Glossary

[0047] Incremental data: data change of the database after a certain time line.

[0048] DML (Data Manipulation Language): insert, delete, update, etc. Change the table data content operation executed on the data.

[0049] DDL (Data Definition Language): create, alter, etc. Change the operation of the library, table structure.

[0050] SYN_POS: bit position information of incremental synchronization, indicating the position information of the current synchronization event in the source library incremental log. Different databases have different forms, in MySQL, it is represented by binlog file name+position or gtid, in postgreSQL, it is represented by LSN, and in Oracle, it is represented by SCN.

[0051] The present application aims at the problem that after the interruption of incremental data synchronization, the source library executes DDL events, and cannot continue from the breakpoint. A kind of incremental data synchronization breakpoint continuation method and system are provided.

[0052] Embodiment one

[0053] As shown in Figure 1 , a kind of incremental data synchronization breakpoint continuation method provided by the present embodiment, including,

[0054] Initialize incremental synchronization task, parse source library by CDC engine, obtain the incremental change of source library;

[0055] Determine whether the incremental change is DDL event or DML event, if the incremental change is DDL event, parse executed DDL statement, and update table metadata and table meta history data;

[0056] If the incremental change is DML event, obtain table metadata based on table meta history data, and generate corresponding SQL statement according to table metadata;

[0057] Connect target library, target end executes the SQL statement generated by parsing, updates synchronization point after successful execution, and continues to parse incremental change from new point.

[0058] Further, the process of initializing incremental synchronization task includes initially creating incremental synchronization task and restarting incremental synchronization task after the interruption of incremental synchronization task;

[0059] If it is initially creating incremental synchronization task, initialization operation includes: recording the current point information and table metadata information of source library to metadata record, and synchronizing the inventory data of table to be synchronized to target end;

[0060] If it is the incremental synchronization task restarted after the interruption of incremental synchronization task, initialization operation is: reading synchronization point from metadata record, and executing incremental data processing from synchronization point.

[0061] Further, the process of parsing source library by CDC engine to obtain the incremental change of source library includes,

[0062] The CDC engine parses the incremental log of the source database to output a standard format event; the event content of the standard format event includes site information, change type, table name involved in the change, field value before and after the change, and executed SQL information; the change type includes DDL events and DML events.

[0063] Further, if the incremental change is a DDL event, the process of parsing the executed DDL statement and updating the table metadata and table history data includes,

[0064] If the incremental change is a DDL event, the executed DDL statement is parsed to obtain the type of change, the fields involved, the indexes, and the constraints, and then the change is applied to the corresponding table metadata, and the table metadata is updated; the type of change includes create, alter, and drop; the fields involved include the addition, deletion, and renaming of fields;

[0065] Meanwhile, the updated table metadata is recorded in the table history data, and different table metadata versions within a time window are recorded based on the table history data; if no DDL change occurs in a time window, the last two table metadata versions are retained.

[0066] Further, if the incremental change is a DML event, the process of obtaining table metadata based on table history data includes,

[0067] If the incremental change is a DML event, the corresponding version of table metadata is queried in the table history data according to the table name and site information; the table metadata is first obtained from the cache, and if it cannot be obtained from the cache, it is obtained from the external storage, and if it cannot be obtained from the external storage, the task enters an exception; the obtained table metadata is parsed to generate a corresponding SQL statement.

[0068] Further, when the external storage cannot be obtained, the continuous number of times the task enters an exception reaches a preset number or the target end executes the parsed SQL statement, and the process of updating the synchronization site after the execution is successful, and the continuous number of times the exception reaches a preset number, further includes,

[0069] The synchronization task is restarted at a fixed time interval defined by the user, and the synchronization continues from the last successful synchronization point; if the number of exceptions exceeds the preset threshold number within the specified time period, the task is stopped and the task is not restarted.

[0070] Further, the process of parsing the table metadata to generate a corresponding SQL statement includes,

[0071] For a DML event, a corresponding SQL execution statement is generated based on the corresponding version of the table history data.

[0072] Furthermore, the process of the target end executing the parsed and generated SQL statement includes,

[0073] If the DDL event is executed successfully, update the synchronization point and continue resolving incremental changes from the new point;

[0074] If the DDL event fails to execute, the corresponding table data in the table's historical data will be deleted.

[0075] Example 2

[0076] like Figure 1 As shown in this embodiment, an incremental data synchronization breakpoint resumption method records the position of the DDL change and the corresponding historical version of the table metadata during the incremental process. After a failure restart, the corresponding version of the table metadata is retrieved based on the position information. This enables breakpoint resumption of incremental data synchronization even if the source database undergoes DDL changes during the interruption, reducing the processing cost of synchronization failures. Specifically, it includes:

[0077] 1. Initialization

[0078] 1) If this is the first time creating an incremental synchronization task, the initialization operations include: recording the current location information and table metadata information of the source database into the metadata record, and synchronizing the existing data of the table to be synchronized to the target end;

[0079] 2) If the synchronization task is interrupted and then restarted, the initialization operation is as follows: read the synchronization point SYN_POS from the metadata record and start incremental data processing from that point.

[0080] 2. Analyze incremental changes

[0081] The incremental logs of the source database are parsed by the CDC engine, and the output is an event in a standard format, including location information, change type (DML event, DDL event), table name involved in the change, the values ​​of the fields involved in the change before and after the change, and the executed SQL information.

[0082] 3. Update table metadata

[0083] If the incremental change is a DDL event, the executed DDL statement is parsed to obtain the type of change (create, alter, drop, etc.), the fields involved (addition, deletion, or renaming of fields), and the addition or removal of indexes and constraints. Then, the changes are applied to the corresponding table metadata, updating the table metadata `Table_meta`.

[0084] 4. Update table data history

[0085] After the DDL event change occurs, the table metadata Table meta updated in the previous step is recorded in the table meta history data meta history. The table meta history data meta history records different table metadata Table meta versions within a time window (such as 24 hours, which can be configured according to actual conditions). If no DDL event change occurs in a time window, the latest two table metadata Table meta versions are retained.

[0086] 5. Obtain table metadata

[0087] If the incremental change is a DML event, the corresponding version of the table metadata Table meta is queried in the table meta history data meta history according to the table name and the position information. The table metadata Table meta is first obtained from the cache, and if the table metadata Table meta cannot be obtained from the cache, the table metadata Table meta is obtained from the external storage. If the table metadata Table meta cannot be obtained from the external storage, the task enters an exception. The synchronization position and the table metadata version correspond to each other as shown in Figure 3 , for example: if the position of the current event is between SYN POS 1 and SYN POS 2, the corresponding table metadata is Table meta 2.

[0088] 6. Generate a SQL statement

[0089] According to the corresponding version of the Table meta, a corresponding SQL execution statement is generated for the DML event.

[0090] 7. Execute a SQL statement

[0091] The Load module connects the target library to execute the SQL statement generated in the previous step. If the DDL execution fails, the corresponding Table meta in the meta history is deleted.

[0092] The Load module is configured to receive the DML event and the DDL event transmitted by the Select module, then connect the target library to execute, and return the execution result to the Select module. If the execution is successful, an ACK is returned, and if the execution fails, a ROLLBACK is returned.

[0093] 8. Update the synchronization position

[0094] After the Load module executes the SQL statement successfully, an ACK is returned to the Select module, indicating that the synchronization of the change event is successful. The Select module updates the position information in the metadata record, and parses the incremental change from the new position.

[0095] The Select module is used to obtain a specified synchronization point from the metadata record, use the CDC (ChangeData Capture) engine to obtain and parse incremental data changes in the source database in real time starting from the synchronization point, and convert them into corresponding DML (Data Manipulation Language) events or DDL (Data Definition Language) events, and then pass the DML events or DDL events to the Load module; it is also used to receive the synchronization results returned by the Load module and update the metadata record.

[0096] Further optimizations include recording the synchronization point information (SYN_POS) and the table's metadata history data (meta_history) through a metadata recording module. The point information represents the current synchronization progress to the target endpoint; SYN_POS is the unique identifier of an event in the database log, and it monotonically increases. Different database types have different representations of SYN_POS: in MySQL, it's represented by the binlog filename + position or gtid; in PostgreSQL, it's represented by LSN; and in Oracle, it's represented by SCN. The metadata history data (meta_history) records the table's metadata version history and consists of two parts: cached and persistent external storage.

[0097] The recording method is: scheme_name.table_name:(SYN_POS_1:Table_meta_1, timestamp:t1),(SYN_POS_2:Table_meta2, timestamp:t2),...,(SYN_POS_n:Table_meta_1, timestamp:tn);

[0098] Where Table_meta_n represents the metadata of table scheme_name.table_name at the SYN_POS_n site, including but not limited to the field information, primary key information, index information and constraint information contained in the table, and tn represents the timestamp corresponding to the site SYN_POS_n.

[0099] 9. Automatic pull-up

[0100] If an exception occurs during synchronization, the program attempts to restart the synchronization task at a fixed time interval T_INTERVAL (customizable, e.g., 10 minutes), continuing synchronization from the last successfully synchronized point. If more than N (customizable, e.g., 6 times) exceptions occur consecutively within a specified time period T (customizable, e.g., 60 minutes), the task stops and will not be restarted.

[0101] The embodiment maintains metadata records in the synchronization process, records the synchronization position and the table metadata of the corresponding version, and can automatically pull up the abnormal task after synchronization exception, reduces the probability of manual operation and maintenance, realizes the breakpoint resume of the incremental data synchronization during interruption, and greatly reduces the cost of synchronization fault processing.

[0102] The method provided by the embodiment can quickly resume the synchronization task from the breakpoint after synchronization exception in the case of real-time synchronization, does not need manual intervention even if the source library table structure is changed, reduces the processing difficulty of data synchronization exception, and thus reduces the operation and maintenance cost.

[0103] Embodiment three

[0104] As shown in Figure 2 The embodiment also provides an incremental data synchronization breakpoint resume system, which comprises:

[0105] A Select module is configured to obtain a specified synchronization position from the metadata records, use a CDC (Change Data Capture) engine to obtain and parse incremental data changes of the source library from the synchronization position in real time, convert the incremental data changes into corresponding DML (Data Manipulation Language) events or DDL (Data Definition Language) events, and then deliver the DML events and the DDL events to a Load module; and the Select module is also configured to receive a synchronization result returned by the Load module and update the metadata records.

[0106] The Load module is connected with the Select module and is configured to receive the DML events and the DDL events delivered by the Select module, then execute connection target libraries, and return an execution result to the Select module; the Load module returns an ACK when the execution is successful, and returns a ROLLBACK when the execution fails.

[0107] The metadata record module is connected with the Select module and is configured to record the position information SYN_POS of synchronization and the table meta history data meta_history.

[0108] Further, the position information is progress information that has been synchronized to the target end at present.

[0109] The SYN_POS is a position identifier of a certain event in a log in a database, the SYN_POS has uniqueness and is monotonically increasing; the SYN_POS in different types of databases has different forms, wherein the SYN_POS in MySQL is represented by the file name + position or gtid of binlog, the SYN_POS in postgresql is represented by LSN, and the SYN_POS in Oracle is represented by SCN.

[0110] The table meta history data meta_history is used for recording the metadata version history of the table, and is divided into two parts of cache and persistent external storage;

[0111] The recording mode is: scheme_name.table_name:(SYN_POS_1:Table_meta_1, timestamp:t1), (SYN_POS_2:Table_meta2, timestamp:t2), …, (SYN_POS_n:Table_meta_1, timestamp:tn);

[0112] Wherein, Table_meta_n represents the metadata of the table scheme_name.table_name at the SYN_POS_n site, including but not limited to the field information, primary key information, index information and constraint information contained in the table, and tn represents the time stamp corresponding to the time point of the site SYN_POS_n.

[0113] In the embodiment, the site after the DDL change and the corresponding table metadata history version are recorded in the incremental process, and after the fault restart, the corresponding version of the table metadata is found according to the site information, so that the incremental data synchronization is realized during the interruption, even if the source library has DDL change, the breakpoint resume can also be realized, and the processing cost of synchronization fault is reduced.

[0114] The incremental data synchronization breakpoint resume system of the embodiment can realize the breakpoint resume of real-time data synchronization, and can also realize the breakpoint resume when the source library table structure is changed during the synchronization interruption, so that the processing difficulty of data synchronization exception is reduced, thereby reducing the operation and maintenance cost.

[0115] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An incremental data synchronization resuming method, characterized in that, The method comprises the following steps: initializing an incremental synchronization task, parsing a source database through a CDC engine to obtain incremental changes of the source database; judging whether the incremental changes are DDL events or DML events, if the incremental changes are DDL events, parsing executed DDL statements and updating table metadata and table history metadata; if the incremental changes are DML events, obtaining table metadata based on the table history metadata and generating corresponding SQL statements according to the table metadata; connecting a target database, executing the generated SQL statements on the target side, updating a synchronization point after successful execution, and continuing to parse incremental changes from the new point; if the incremental changes are DDL events, the process of parsing executed DDL statements and updating table metadata and table history metadata comprises: if the incremental changes are DDL events, parsing executed DDL statements, obtaining the type of changes, involved fields, indexes and constraints, and then applying the changes to the corresponding table metadata and updating the table metadata; wherein the type of changes includes create, alter and drop; the involved fields include addition, deletion and renaming of fields; at the same time, recording the updated table metadata in the table history metadata, recording different table metadata versions within a time window based on the table history metadata, and if no DDL changes occur within a time window, retaining the latest two table metadata versions; if the incremental changes are DML events, the process of obtaining table metadata based on the table history metadata comprises: if the incremental changes are DML events, querying the corresponding version of table metadata in the table history metadata according to the table name and the point information, first obtaining from the cache, if not obtained from the cache, obtaining from the external storage, if not obtained from the external storage, the task enters an exception, parsing the obtained table metadata to generate corresponding SQL statements; when the number of consecutive times of entering an exception reaches a preset number or an exception occurs in the process of executing the generated SQL statements on the target side and updating the synchronization point, and the number of consecutive times of entering an exception reaches a preset number, further comprising: trying to restart the synchronization task at a fixed time interval, continuing synchronization from the last successful synchronization point, if the number of consecutive exceptions within a specified time period exceeds a preset threshold, the task is stopped and the task is not restarted.

2. The incremental data synchronization breakpoint continuation method according to claim 1, wherein: the process of initializing the incremental synchronization task comprises initially creating the incremental synchronization task and restarting the incremental synchronization task after interruption; if the incremental synchronization task is initially created, the initialization operation comprises recording the current point information and table metadata information of the source database in the metadata record, and synchronizing the inventory data of the table to be synchronized to the target side; if the incremental synchronization task is restarted after interruption, the initialization operation comprises reading the synchronization point from the metadata record and starting incremental data processing from the synchronization point.

3. The incremental data synchronization breakpoint resuming method according to claim 1, characterized in that, the process of obtaining the incremental changes of the source database by parsing the source database through the CDC engine comprises, the process of outputting the standard format events by parsing the incremental logs of the source database through the CDC engine comprises that the event content of the standard format events comprises the site information, the change type, the table name involved in the change, the field values before and after the change, and the executed SQL information; and the change type comprises the DDL event and the DML event.

4. The incremental data synchronization breakpoint resuming method according to claim 1, characterized in that, the process of generating the corresponding SQL statements according to the table metadata comprises, for the DML event, the corresponding SQL execution statement is generated according to the table metadata of the corresponding version.

5. The incremental data synchronization breakpoint resuming method according to claim 1, characterized in that, the process of executing the generated SQL statements at the target end comprises, if the DDL event is executed successfully, the synchronization site is updated and the incremental changes are parsed from the new site; if the DDL event fails to be executed, the corresponding table metadata in the table metadata history data is deleted.

6. An incremental data synchronization resume system, comprising: comprises: a Select module configured to obtain the specified synchronization site from the metadata record, parse the incremental data changes of the source database from the synchronization site in real time by using the CDC engine, convert the incremental data changes into the corresponding DML event or DDL event, and then pass the DML event and the DDL event to a Load module; and further configured to receive the synchronization result returned by the Load module and update the metadata record; the Load module connected to the Select module, configured to receive the DML event and the DDL event passed by the Select module, then execute the DML event and the DDL event in connection with a target database, and return the execution result to the Select module, return ACK if the execution is successful, and return ROLLBACK if the execution fails; a metadata record module connected to the Select module, configured to record the site information SYN POS and the table metadata history data of the table; if the incremental change is the DDL event, the process of parsing the executed DDL statement and updating the table metadata and the table metadata history data comprises, if the incremental change is the DDL event, the type of the change, the fields involved, the indexes, and the increase and decrease of the constraints are obtained by parsing the executed DDL statement, then the change is applied to the corresponding table metadata, and the table metadata is updated; wherein the type of the change comprises create, alter, and drop; and the fields involved comprise the addition, deletion, and renaming of the fields; at the same time, the updated table metadata is recorded in the table metadata history data, and different table metadata versions within a time window are recorded based on the table metadata history data; if no DDL change occurs in the time window, the latest two table metadata versions are retained; if the incremental change is the DML event, the process of obtaining the table metadata based on the table metadata history data comprises, If the incremental change is a DML event, the corresponding version of the table metadata is queried in the table metadata history data according to the table name and the position information, is obtained from the cache first, is obtained from the external storage if not obtained from the cache, and the task enters an exception if not obtained from the external storage. The obtained table metadata is parsed to generate a corresponding SQL statement. When the task enters an exception and the number of continuous exceptions reaches a preset number or the target end executes the parsed SQL statement, an exception occurs in the process of updating the synchronization position after the execution is successful, and the number of continuous exceptions reaches a preset number, the method further includes: restarting the synchronization task at a self-defined fixed time interval, continuing synchronization from the last synchronization success position, and stopping the task if the number of continuous exceptions exceeds a preset threshold number within a self-defined specified time period.

7. The incremental data synchronization breakpoint continuation system of claim 6, wherein, the position information is progress information that has been synchronized to the target end; SYN POS is a position identifier of a certain event in a log in a database, SYN POS has uniqueness and is monotonically increasing; the table metadata history data is used for recording a metadata version history of a table and is divided into a cache and persistent external storage; a recording mode is: scheme_name.table_name: (SYN_POS_1: Table_meta_1, timestamp: t1), (SYN_POS_2: Table_meta2, timestamp: t2), …, (SYN_POS_n: Table_meta_1, timestamp: tn); wherein, Table_meta_n represents metadata of the table scheme_name.table_name at the position SYN_POS_n, including but not limited to field information, primary key information, index information and constraint information of the table, and tn represents a timestamp of a time corresponding to the position SYN_POS_n.

Images