Method and system for real-time incremental synchronization of heterogeneous databases for migration to cloud

Abstract

The present application relates to the technical field of data migration and synchronization, in particular to a real-time incremental synchronization method and system for migrating heterogeneous databases to the cloud, which can realize real-time capture of change log streams of local source databases through orchestration plugins, parse and reconstruct the change log streams into a unified event change model, complete data processing capabilities such as field mapping and type conversion, realize table structure synchronization, writing to the target database in the cloud, and processing and fault tolerance mechanisms for abnormal situations, and realize automatic scheduling through a DSServer scheduler, thereby realizing efficient, stable and reliable synchronization of cross-platform database migration to the cloud, realizing low-latency and high-consistency data synchronization from the source database to the target database, ensuring smooth transition of the business system during the migration process, and being especially suitable for microservice architecture environments of large and complex systems such as power grids.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data migration and synchronization, in particular to a real-time incremental synchronization method and system for migrating heterogeneous databases to the cloud. BACKGROUND

[0002] With the development of cloud computing, more and more enterprises migrate local databases to cloud database platforms. Due to structural differences between source databases and target databases, data type incompatibility and other problems, traditional full backup recovery or periodic batch synchronization methods cannot meet the requirements of high availability, low latency and data consistency.

[0003] In the power grid system, the business system is complex, the data volume is large, and the real-time, consistency and system availability of data synchronization are extremely high. With the wide application of micro-service architecture, multiple business modules in the power grid system have been deployed as services, and data is distributed in multiple heterogeneous databases. In order to meet the evolution trend of cloud-native architecture, how to realize seamless migration of local heterogeneous databases to cloud databases under the premise of continuous operation and uninterrupted business, and guarantee smooth switching of the business after migration, has become a key technical problem to be solved.

[0004] Currently, there are many deficiencies in the synchronization process of heterogeneous databases: on the one hand, manual programming of field mapping and type conversion rules is required, which has high maintenance cost, and commercial tools lack support for logical deletion and DDL synchronization, which requires customized development; on the other hand, the synchronization scheme based on triggers invades the business and has large performance loss, and open source tools do not realize automatic type conversion and dynamic field mapping.

[0005] Therefore, a technical solution with strong universality, high flexibility and supporting real-time synchronization of multiple source heterogeneous databases is urgently needed, especially for large and complex systems such as power grids that have extremely high requirements for data synchronization. SUMMARY

[0006] The present application aims to provide a real-time incremental synchronization method and system for migrating heterogeneous databases to the cloud, to realize efficient, stable and reliable synchronization of cross-platform database migration, meet the requirements of low latency and high consistency of data synchronization, and guarantee smooth transition of the business system.

[0007] The present application solves the above technical problems by the following technical scheme: a real-time incremental synchronization method for migrating heterogeneous databases to the cloud, the method comprising the following steps:

[0008] S1, capturing the change log stream of the local source database in real time through the SourceDump plug-in;

[0009] S2, the change log stream is parsed and reconstructed into a unified event change model by a Transform_UCEF plug-in, and redundant information in the change stream is processed by a change stream compression algorithm, and only field information that has changed is recorded;

[0010] S3, data type conversion, field mapping, adding a logical deletion mark and adding a synchronization timestamp mark are realized by a FieldMap plug-in;

[0011] S4, in the Transform_UCEF plug-in, a DDL type event is responded to, table structure synchronization is realized, only DDL events of operation types of ALTER TABLE and CREATE TABLE are processed, are dynamically mapped and converted into DDL statements suitable for a target database, and corresponding table structures are automatically created or modified in the target database;

[0012] S5, a change event is written into a cloud target library and a synchronization site is recorded by a CopyTarget plug-in;

[0013] S6, an exception handling and fault tolerance mechanism is packaged as a plug-in process, the exception handling and fault tolerance mechanism includes full and incremental collaborative consistent data verification in a data migration process, and corresponding processing measures are taken for the cases of data inconsistency, target library unavailability, network and other abnormal transmission interruptions;

[0014] S7, the plug-in process arranged is automatically scheduled by a DSServer scheduler.

[0015] On the other hand, the application also provides a heterogeneous database real-time incremental synchronization system for migrating to the cloud, comprising:

[0016] A SourceDump plug-in module is used for capturing a source database change log stream;

[0017] A Transform_UCEF plug-in module is used for parsing a log and reconstructing into a unified event model, and realizing table structure synchronization in response to a DDL event;

[0018] A FieldMap plug-in module is used for executing data type conversion, field mapping and mark addition;

[0019] A CopyTarget plug-in module is used for writing into a target library and recording a synchronization site;

[0020] An exception handling and fault tolerance plug-in module is used for executing data verification and exception handling;

[0021] A plug-in arrangement module is used for providing a visual arrangement function;

[0022] DSSERVER scheduling module: used for realizing plug-in process automatic scheduling.

[0023] The positive progress effect of the application is that:

[0024] 1. The application supports high-throughput (>=100,000 pieces / sec), low-delay (<=100ms) data synchronization, meets the high requirement of real-time of business, has real-time exception processing and breakpoint resume ability, and the fault recovery time is less than 1 minute, so as to guarantee the stability and reliability of data synchronization.

[0025] 2. The system of the application supports automatic conversion of data types and dynamic mapping of fields between mainstream heterogeneous databases, reduces manual intervention, significantly reduces operation and maintenance cost, adopts plug-in and visual arrangement design, is high in flexibility, can be quickly adapted and adjusted according to different business scenes, is strong in applicability, is especially suitable for large and complex systems such as power grids which have very high requirements for data synchronization, and provides efficient and reliable technical support for cloud migration of the databases. BRIEF DESCRIPTION OF DRAWINGS

[0026] Fig. 1 The system architecture schematic diagram provided by the application is shown.

[0027] Fig. 2 The unified event change model structure diagram provided by the application is shown.

[0028] Fig. 3 The exception processing and fault tolerance mechanism schematic diagram provided by the application is shown. DETAILED DESCRIPTION

[0029] The application will be further described by way of examples below, but the application is not limited in the scope of the examples.

[0030] Referring to Figs. 1-3 The application provides a heterogeneous database real-time incremental synchronization method for cloud migration, taking the migration of local Mysql database data to cloud pg database as an example, which includes the following steps:

[0031] S1, capture change log stream: the source database includes MySQL and PostgreSQL, and the change log stream is binlog log of MySQL and wal log of PostgreSQL respectively; enable the SourceDump plug-in, select the source database type as MySQL in the plug-in configuration option, and transmit the captured binlog log to the Transform_UCEF plug-in through the SourceDump plug-in to monitor the binlog log generation of the MySQL source database in real time.

[0032] S2, parsing and reconstructing into a unified event model: the Transform_UCEF plug-in parses the binlog log after receiving it, listens to TableMapEvent to obtain the table structure, parses WriteRows EventData, UpdateRowsEventData, and DeleteRows EventData to obtain the data changes of the insertion, update, and deletion operations respectively, parses the DDL statement in QueryEvent to extract the table structure change information, processes the redundant information through a change stream compression algorithm, records only the changed fields, integrates the operation type, table name, field data, primary key, original data value, new data value, changed value, timestamp, and other information, and reconstructs into a unified event change model;

[0033] Because the display in the change log stream of each database is different, the process of log parsing operation is customized for different databases, and a unified event change model is constructed. For example: MySQL database connects through binary log, specifies binlog file and position to start listening; parse TableMapEvent to get table metadata, cache database name, table name, column definition, data type; data change processing includes: insert event: parse WriteRowsEventData to extract new row data; update event: parse UpdateRowsEventData to extract data before and after change; delete event: parse DeleteRowsEventData to extract deleted row data; process DDL change by parsing SQL statement in QueryEvent, identify CREATE / ALTER / DROP DDL operation, extract table structure change information, and convert all events to standard data model. If PostgreSQL database uses pgoutput plug-in, logical replication slot can be created to establish logical decoding slot for continuous capture of database changes; poll WAL changes, regularly extract binary format WAL log records from replication slot; parse binary data according to pgoutput protocol, identify message type (including Begin / Commit / Relation / Insert / Update / Delete / Truncate); extract DDL operation information, parse table structure change from Relation message, including CREATE / ALTER / DROP table DDL operation, get complete table structure information, including column definition, data type, constraint, etc.; convert DDL and DML changes into standardized data model, including operation type, schema, table name, table structure, data change, etc. For a large amount of redundant information in the change stream, change stream compression algorithm can be used to process the information in the change stream, record the changed part in the change model, and clearly record which fields are changed each time, which is convenient for state rollback and change history query:

[0034] ;

[0035] ;

[0036] ;

[0037] wherein, represents the difference between the data state at time t and time t-1, is the data state at time t, is the data state at time t-1, The changed field set, f is a field in all fields in the data model, id is the unique identifier of the change event, op is the operation type of the change, timestamp is the timestamp of the change, Other metadata.

[0038] As above, the unified event change model includes DML operation types (add, delete, change) and DML operation types, timestamps, table names, field information, primary keys, original data values, new data values, and change values. The way of parsing and reconstructing the change log stream into a unified event change model is encapsulated in the Transform_UCEF plug-in, and the parsing type item can be configured, including DDL events and DML events, wherein, Indicates the difference between the data state at time t and the data state at time t-1, The data state at time t, The data state at time t-1, The changed field set, f is a field in all fields in the data model, id is the unique identifier of the change event, op is the operation type of the change, timestamp is the timestamp of the change, Other metadata.

[0039] The change stream compression algorithm further combines the pre-analyzed data distribution characteristics and business rules in the metadata knowledge base; for wide tables with high-frequency updates and only a few field changes, the algorithm uses a primary key-based incremental snapshot comparison; for business data with time series or version number characteristics, the algorithm can identify and only synchronize the latest valid version, achieving business semantic level compression and further reducing network transmission and target library write load.

[0040] S3, data processing: transmit the unified event change model to the FieldMap plug-in, the configurable items of the FieldMap plug-in include field mapping rules, data type conversion rules, whether to add a logical deletion marker and the corresponding field, whether to add a synchronization timestamp marker and the corresponding field, in the plug-in configuration, the data type mapping rules of MySQL and PostgreSQL are preset, the field mapping rules, the logical deletion marker and the synchronization timestamp marker are enabled, the plug-in completes data type conversion and field mapping according to the configuration rules, adds a logical deletion marker to the event of the DELETE operation, so that the record can be identified and processed as a logical deletion when output to the target database, and adds the current system time as a synchronization timestamp in each event model, which is used for subsequent synchronization time recording and monitoring;

[0041] According to the data type mapping relationship of the source database and the target database, the data in the unified event change model is subjected to data type conversion, so as to ensure that the data can be correctly stored in the target database. Meanwhile, according to the configuration rule, the fields in the source database are mapped to the corresponding fields of the target database, so as to adapt to the table structure of the target database. In addition, in order to prevent the case of mistakenly deleting data, a logical deletion mark can be added in the event change model, and when a deletion operation occurs in the source database, the record is marked as logical deletion in the event. Meanwhile, a synchronization timestamp can be added, which records the time of data synchronization, so as to facilitate subsequent monitoring and tracing. The data type conversion and the field mapping mode are encapsulated in the FieldMap plug-in, the field mapping and its type can be configured, whether the synchronization timestamp and its field are needed can be configured, whether the logical deletion and its field are needed can be configured; the FieldMap plug-in integrates the mapping rule automatic derivation function of the metadata driven; in the case of lacking explicit configuration, the plug-in can call the mapping knowledge base, based on the field name similarity, the data type compatibility and the pattern recognition of the sample data value, to automatically generate a candidate mapping rule for the user to confirm or directly apply, which greatly reduces the initialization configuration workload of the heterogeneous database synchronization.

[0042] S4, table structure synchronization: responding to the DDL type event in the Transform_UCEF plug-in, the table structure synchronization is realized, and the conversion DDL item is configured in the Transform_UCEF plug-in, so that the plug-in can monitor the table structure change of the MySQL source database in real time during running;

[0043] The table structure change of the source database is monitored in real time, including the addition, deletion and modification of the field structure. When the unified event change model after log parsing is detected to have a table structure change, that is, the operation type is ALTER TABLE and CREATE TABLE, other operations such as DROP TABLE are ignored. The DDL statement suitable for the target database is dynamically mapped and converted; the corresponding table structure is automatically created or modified in the target database, so as to ensure that the table structure of the target database is consistent with that of the source database, so as to ensure that the data can be correctly written into the target database. If it is CREATE TABLE, the table structure information in the unified data model is directly obtained, the type mapping of the corresponding target table is constructed, and the DDL table creation statement of the target table is formed. If it is ALTER TABLE, the DDL statement needs to be decomposed, the specific operation type, table name, annotation and other information in the ALTER are obtained, the type mapping of the corresponding target table is constructed, and the DDL modification statement of the target table is formed. The responding DDL type event mode is encapsulated in the Transform_UCEF plug-in, and the parsing type item can be configured, including the DDL event and the DML event;

[0044] In addition to basic table structure mapping, the Transform_UCEF plug-in can handle adaptive conversion of partitioned tables, indexes, and constraints (such as foreign keys and unique constraints) based on the mapping knowledge base when processing DDL events. For cloud-native target database-specific features such as distributed sharding keys and TTL settings, the plug-in can automatically recommend and adapt based on policy templates. Meanwhile, for cascading DDL operations involving multiple tables (such as modifying associated field types), the plug-in can identify the associations and coordinate the synchronization order of multiple table structures to ensure transaction consistency.

[0045] S5, Write target library and record synchronization site: The processed unified event change model is received by the CopyTarget plug-in, and insert, update, or delete statements are generated according to the PostgreSQL SQL syntax. Based on the field mapping rules of the FieldMap plug-in and other configuration items, it is written into the PostgreSQL target database. During the writing process, the current processing position in the MySQL binlog (such as file name and offset) is recorded as the synchronization site, and this site information is stored in the built-in library.

[0046] The CopyTarget plug-in supports batch submission and transaction control to improve the performance of writing to the cloud target library. The synchronization site is stored in the built-in library for fault recovery and breakpoint resume after system failure or interruption.

[0047] The processed unified event change model is written into the target database (such as MySQL, PostgreSQL, Oracle, etc.) according to the writing requirements of the target database. Batch submission and transaction control are supported to improve writing performance. The synchronization site (checkpoint) is recorded after each successful write, which is the current synchronization position to the source database change log (such as binlogposition, LSN), for fault recovery and breakpoint resume. After the system fails or interrupts, it can continue synchronization from the synchronization site to ensure the continuity and reliability of data synchronization. Writing to the cloud target library and recording the synchronization site are encapsulated in the CopyTarget plug-in, and the target library item can be configured.

[0048] S6, Exception handling and fault tolerance: When the network is interrupted, the local cache component automatically writes the unsynchronized events to the local disk, with a single cache file size limit of 1GB, and timestamp-based sharding storage. After the network is restored, the system obtains the last successful synchronization site from the built-in library, and the SourceDump plug-in starts to capture the binlog log from this site. The subsequent plug-ins process according to the process to realize breakpoint resume.

[0049] The abnormality processing and fault tolerance plug-in module performs incremental consistency check regularly, extracts change records in the past one hour based on the synchronization point, and triggers an automatic compensation process if the check component detects that a record in a table has inconsistent values in the same field in MySQL and PostgreSQL: the complete data of the record is re-pulled from MySQL, a compensation event is generated and marked as "high priority", the error value is overwritten in PostgreSQL, and the record is logged after repair. The abnormality processing and fault tolerance mechanism in S6 further includes intelligent difference detection based on data profiling. The system uses the field importance classification (e.g., primary key field, financial amount field, and state field are high sensitive levels) stored in the mapping knowledge base to perform different check frequencies and accuracies on different levels of fields during incremental check. For high sensitive fields, real-time row-level check is adopted. For low sensitive fields, sampling check is adopted. When inconsistency is detected, intelligent root cause analysis is performed based on the change event flow context and field characteristics to distinguish whether it is network packet loss, plug-in logic error, or manual error operation on the target side, and a differentiated recovery strategy is adopted accordingly to improve fault location and recovery efficiency.

[0050] The abnormality processing and fault tolerance mechanism is packaged as a plug-in process. The abnormality processing and fault tolerance mechanism performs full and incremental data check during data migration. After the initialization synchronization is completed, full consistency check is performed to establish the data baseline of the source database and the target database. At the same time, full comparison can also be performed through hash check and other methods at a low business peak period every day. During subsequent incremental migration, incremental consistency check is performed based on the synchronization point on the basis of the data baseline to monitor near real-time data changes. The entire abnormality processing and fault tolerance mechanism is packaged as a plug-in process, and the check method is configurable. After the data migration initialization is completed, full consistency check is performed to establish the data baseline of the source database and the target database. Incremental consistency check is performed based on the synchronization point to monitor near real-time data changes. Specific abnormalities and their processing methods include:

[0051] Data inconsistency: difference records are located based on primary key hash value, automatic compensation synchronization (data is re-pulled from the source database) is initiated, and difference logs are recorded to the meta database; if inconsistency is caused by data structure processing error, an alarm is triggered to require manual processing.

[0052] Target library unavailability: write is suspended and change events are cached, and after the target library is restored, the synchronization is continued according to the synchronization point, and the cache capacity supports ≥2 hours of peak data volume;

[0053] Network and other abnormalities cause transmission interruption: local log cache is enabled, the breakpoint is automatically checked after network recovery, and incremental synchronization mode is used to supplement the data during the interruption period.

[0054] S7, plug-in visual arrangement and automatic scheduling: in the designer, by visual drag and drop, the SourceDump plug-in, Transform_UCEF plug-in, FieldMap plug-in, CopyTarget plug-in and exception handling and fault tolerance plug-in are arranged in the above process sequence, the parameters of each plug-in are configured, the arranged plug-in process is submitted to DSServer scheduler, the scheduling period is set to real-time trigger, the scheduler automatically monitors the source database change, triggers the plug-in process execution, and realizes automatic synchronization;

[0055] The plug-in is realized by visual arrangement, and the DSSERVER scheduler is realized by automatic scheduling, so that each of the above points is made into a plug-in that can arrange the process and task in the designer, and the plug-in has configurable items. Through visual drag and drop arrangement in the designer, incremental task configuration according to requirements is realized, and flexibility is increased. And automatic scheduling can be carried out through DSServer scheduler, reducing human intervention; the DSServer scheduler has elastic scheduling capability of load sensing; real-time monitoring of each plug-in processing queue depth, CPU / memory usage and network IO; when it is detected that the capture rate of the SourceDump plug-in is continuously higher than the processing capacity of the subsequent plug-in, the Transform_UCEF or CopyTarget plug-in can be automatically expanded to process instances; otherwise, it is contracted to save resources. In addition, the scheduler can perform predictive scheduling according to historical traffic rules, pre-expand before the business peak, and ensure synchronization stability.

[0056] S8, migration verification: after data migration initialization is completed, the exception handling and fault tolerance plug-in module performs full consistency verification, performs key field hash verification on the core business table, ensures that the hash consistency rate reaches 100%, and establishes a data baseline.

[0057] In addition, the application also provides a heterogeneous database real-time incremental synchronization system for migrating to the cloud, comprising:

[0058] The SourceDump plug-in module is used for capturing the source database change log stream;

[0059] The Transform_UCEF plug-in module is used for parsing the log and reconstructing into a unified event model, and responding to the DDL event to realize table structure synchronization;

[0060] The FieldMap plug-in module is used for performing data type conversion, field mapping and tag addition;

[0061] The CopyTarget plug-in module is used for writing into the target database and recording the synchronization point;

[0062] The exception handling and fault tolerance plug-in module is used for performing data verification and exception handling;

[0063] Plugin orchestration module: for providing visual orchestration function;

[0064] DSSERVER scheduling module: for realizing automatic scheduling of plugin process.

[0065] The change log stream of the local source database is captured in real time through the SourceDump plugin. The generation of the binlog log of the MySQL source database is monitored in real time through the SourceDump plugin, and the captured binlog log is transmitted to the Transform_UCEF plugin. For the PostgreSQL target database, there is no need to perform log capture on the target side, and the main concern is the log capture of the source database MySQL.

[0066] Embodiment one:

[0067] The change log stream is parsed and reconstructed into a unified event change model through the Transform_UCEF plugin. In the Transform_UCEF plugin, the binlog log of MySQL is parsed, and the operation type (such as INSERT, UPDATE, DELETE, including ALTER TABLE DDL operation type), table name, field list and corresponding data value of each change record are extracted. Then, according to the preset unified event change model format, such as Fig. 2 as shown, these information is reconstructed into a unified event model containing operation type, table name, field data, logical deletion flag (initially not deleted), synchronization timestamp (current time).

[0068] Data type conversion and field mapping are realized through the FieldMap plugin. In the FieldMap plugin, the following each item is sequentially.

[0069] Data type conversion: due to the differences in data types between MySQL and PostgreSQL, for example, the BIGINT type of MySQL corresponds to the INT8 type of PostgreSQL, in the data processing module, according to the preset data type mapping table, the data type of MySQL is converted to the corresponding data type of PostgreSQL.

[0070] Field mapping: if the table field name in the source database MySQL is inconsistent with the table field name in the target database PostgreSQL, through field mapping configuration, the source field name is mapped to the target field name, to ensure that the data can be correctly written into the corresponding field of the target table.

[0071] Logical deletion mark: When parsing the DELETE operation in the binlog, a logical deletion mark can be added to identify and handle the record as logical deletion when output to the target database.

[0072] Synchronous timestamp: Add the current system time as a synchronous timestamp in each event model for subsequent synchronization time recording and monitoring.

[0073] In the Transform_UCEF plugin, respond to DDL type events to achieve table structure synchronization. Configure the transform DDL item in the Transform_UCEF plugin, which will monitor the table structure changes of the MySQL source database in real time during runtime. For example, when a new field "name" is added to the MySQL table table, the operation type ALTER TABLE is detected in the model after log parsing, the entire operation statement is identified and decomposed, the specific change type ADD COLUMN is extracted, the table name is table, the field name is name, and the field type is varchar(255), the type mapping of the corresponding PostgreSQL target table is constructed, which is also varchar(255). Integrate to form a new table structure modification statement and add the corresponding "name" field in the target table.

[0074] The CopyTarget plugin writes change events to the cloud target library and records the synchronization point. The CopyTarget plugin generates insert, update, or delete statements according to the PostgreSQL SQL syntax based on the unified event model processed by the Transform_UCEF plugin, and writes them to the PostgreSQL target database according to the field mapping rules of the FieldMap plugin and other configuration items. During the writing process, record the current processing position of the MySQL binlog (such as file name and offset) as the synchronization point, and store the point information in the built-in library.

[0075] Exception handling and fault tolerance mechanism.

[0076] Example two:

[0077] In the case of network interruption, the local cache component automatically writes unsynchronized events to the local disk, with a single file size limit of 1GB and timestamp sharding. After network recovery, the system obtains the last successful point from the synchronization point management module and executes steps from (1) to sequentially supplement the transmission.

[0078] Example three:

[0079] In the case of data inconsistency, the checking component detects that a record in a certain table has inconsistent values in the same field in MySQL and PostgreSQL, triggers an automatic compensation process: reloads the complete data of the record from MySQL, generates a compensation event and marks it as "high priority", synchronizes it to PostgreSQL to overwrite the error value, and logs the record after repair.

[0080] Migration verification

[0081] Full volume verification: After the initialization synchronization is completed, the hash verification of the key fields of the core business table is performed (the hash consistency rate is 100%).

[0082] Incremental verification: Based on the synchronization point, the change record of "last hour" is extracted every hour, and the final state of the source end and the target end is compared to ensure that the inconsistency rate is less than or equal to 0.001%.

[0083] The method of the above data migration is realized by a plug-in to realize visual drag-and-drop arrangement, the abnormality processing and the fault tolerance mechanism verification are encapsulated as one of the plug-in processes, the scheduling automation is realized through a DSSERVER scheduler, the scheduler can be executed regularly or according to the conditions in the verification process, and human intervention is reduced.

[0084] The present application is not limited to the above-mentioned embodiments, and any changes in shape or structure fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims, and those skilled in the art can make various changes or modifications to the embodiments without departing from the principles and essence of the present application. However, these changes and modifications fall within the scope of protection of the present application.

Claims

1. A method for real-time incremental synchronization of heterogeneous databases for migration to cloud, characterized in that, The method comprises the following steps: S1, capturing the change log stream of the local source database in real time through a SourceDump plug-in; S2, parsing and reconstructing the change log stream into a unified event change model through a Transform_UCEF plug-in, and processing redundant information in the change stream through a change stream compression algorithm, and only recording the field information that has changed; The unified event change model in S2 comprises a DML operation type, a DDL operation type, a timestamp, a table name, field information, a primary key, an original data value, a new data value, and a change value; The specific process of parsing and reconstructing the change log stream in S2 is as follows: For a MySQL database, the TableMapEvent is listened to through a MySQL replication stream to obtain the table structure, the WriteRows EventData, the UpdateRows EventData, and the DeleteRows EventData are parsed to obtain data changes, the DDL statement in the QueryEvent is parsed to extract the table name, the operation type, the column definition, and the data change, and the unified model is converted; For a PostgreSQL database, the change is obtained through the creation of a logical replication slot, the binary data is parsed according to the protocol followed by the logical decoding output plug-in, the message type is identified, the table structure change is parsed from the Relation message, including the creation and modification of the table DDL operation, and the DDL and DML changes are uniformly converted into a standardized data model; S3, realizing data type conversion, field mapping, adding a logical deletion marker, and adding a synchronization timestamp marker through a FieldMap plug-in; S4, responding to the DDL type event in the Transform_UCEF plug-in to realize table structure synchronization, only processing the DDL events of the operation type of ALTER TABLE and CREATE TABLE, dynamically mapping and converting into a DDL statement suitable for the target database, and automatically creating or modifying the corresponding table structure in the target database; S5, writing the change event into the cloud target library through a CopyTarget plug-in and recording the synchronization site; S6, packaging the exception handling and fault tolerance mechanism into a plug-in process, wherein the exception handling and fault tolerance mechanism comprises full and incremental collaborative consistent data verification during data migration, and corresponding processing measures are taken for the cases of data inconsistency, target library unavailability, network and other abnormal transmission interruptions; S7, automatically scheduling the arranged plug-in process through a DSServer scheduler.

2. The method for real-time incremental synchronization of heterogeneous database for migration to cloud as claimed in claim 1 wherein, The source database in S1 comprises at least MySQL and PostgreSQL, and the corresponding change log streams are binlog logs of MySQL and wal logs of PostgreSQL.

3. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 1 wherein, The formula for realizing change information processing in the change stream compression algorithm in S2 is: ； ； ； wherein, represents the difference between the data state at time t and time t-1, is the data state at time t, is the data state at time t-1, represents the set of fields that have changed, f is a field from the set of all fields fields in the data model, id is a unique identifier for the change event, op is the type of operation that changed, timestamp is the timestamp at which the change occurred, other metadata.

4. The method for real-time incremental synchronization of heterogeneous database for migration to cloud as claimed in claim 2 wherein: The configurable items of the FieldMap plug-in in S3 comprise field mapping rules, data type conversion rules, whether to add a logical deletion marker and the corresponding field, whether to add a synchronization timestamp marker and the corresponding field.

5. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 1 wherein: In the S4, the table structure information in the unified data model is directly obtained when the CREATE TABLE event is processed, the type mapping of the corresponding target table is constructed, and the DDL table creation statement of the target table is formed. In the S5, the target library includes at least MySQL, PostgreSQL and Oracle, and the synchronization point is the position currently synchronized to the source database change log, including the binlog position of MySQL and the LSN of PostgreSQL. In the S5, the CopyTarget plug-in supports batch submission and transaction control, and the synchronization point is stored in the built-in library, which is used for fault recovery and breakpoint continuation after system failure or interruption.

6. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 5 wherein: In the S6, for the case of data inconsistency, the difference record is located based on the primary key hash value, the compensation synchronization is automatically initiated to pull data from the source library, and the difference log is recorded to the metadata database. In the S6, the abnormality processing and fault tolerance mechanism further includes: intelligent difference detection based on data portrait; the system uses the field importance classification stored in the mapping knowledge base to adopt different verification frequencies and accuracies for different level fields during incremental verification; for high sensitive fields, real-time row level verification is adopted; for low sensitive fields, timing sampling verification can be adopted; when inconsistency is detected, intelligent analysis of the root cause is performed in combination with the change event stream context and field characteristics to distinguish whether it is network packet loss, plug-in logic error or target end manual error, and a differentiated recovery strategy is adopted accordingly to improve fault location and recovery efficiency.

7. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 1 wherein: In the S6, for the case that the target library is unavailable, the write is suspended and the change events are cached, the cache capacity supports ≥2 hours of peak data volume, and after the target library is restored, the synchronization is continued according to the synchronization point; For the case of network and other abnormalities leading to transmission interruption, local log cache is enabled, the breakpoint is automatically verified after network recovery, and incremental synchronization mode is adopted to supplement the data during the interruption period.

8. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 1 wherein: In the S7, the visual drag-and-drop arrangement function of the designer supports flexible adjustment of plug-in combination order and plug-in configuration items according to different business requirements to adapt to the heterogeneous database incremental synchronization requirements in different scenarios. The SourceDump plug-in module is used to capture the source database change log stream.

9. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as claimed in claim 1 wherein: The Transform_UCEF plug-in module is used to parse the log and reconstruct the unified event model, and respond to the DDL event to realize table structure synchronization.

10. The system for real-time incremental synchronization of heterogeneous database migration to cloud as claimed in any one of the claims 1 to 9, wherein: The FieldMap plug-in module is used to perform data type conversion, field mapping and tag addition. The CopyTarget plug-in module is used to write to the target library and record the synchronization point. The abnormality processing and fault tolerance plug-in module is used to perform data verification and abnormality processing. The plug-in arrangement module is used to provide visual arrangement function. ​ ​ ​ DSSERVER scheduling module: used for realizing plug-in process automatic scheduling.

Images