Heterogeneous database real-time increment synchronization method and system for cloud migration

Abstract

The invention relates to the technical field of data migration and synchronization, in particular to a heterogeneous database real-time incremental synchronization method and system for cloud migration, which can capture a change log stream of a local source database in real time by arranging a plug-in, analyze and reconstruct the change log stream into a uniform event change model, and improve the data migration and synchronization efficiency. According to the method, data processing capabilities such as field mapping and type conversion are completed, functions such as table structure synchronization, writing into a cloud target database and processing and fault-tolerant mechanism aiming at abnormal conditions are realized, and automatic scheduling is realized through a DSServer scheduler, so that efficient, stable and reliable synchronization of cloud migration on a cross-platform database is realized, and the cloud migration efficiency is improved. Low-delay and high-consistency data synchronization from the source database to the target database is achieved, smooth transition of a service system in the migration process is ensured, and the method is particularly suitable for micro-service architecture environments of large complex systems such as a power grid.

Description

Technical Field

[0001] This invention relates to the field of data migration and synchronization technology, specifically to a method and system for real-time incremental synchronization of heterogeneous databases for migration to the cloud. Background Technology

[0002] With the development of cloud computing, more and more enterprises are migrating their local databases to cloud database platforms. Due to structural differences and data type incompatibility between the source and target databases, traditional full backup and recovery or periodic batch synchronization methods are difficult to meet the requirements of high availability, low latency, and data consistency.

[0003] In power grid systems, business systems are complex, data volumes are massive, and the requirements for real-time data synchronization, consistency, and system availability are extremely high. With the widespread adoption of microservice architecture, multiple business modules in power grid systems have been deployed as services, with data distributed across multiple heterogeneous databases. To meet the evolution trend of cloud-native architecture, how to achieve seamless migration from local heterogeneous databases to cloud databases while ensuring continuous and uninterrupted business operations, and how to guarantee a smooth business transition after migration, has become a critical technical challenge that urgently needs to be addressed.

[0004] Currently, there are many shortcomings in the synchronization process of heterogeneous databases: on the one hand, field mapping and type conversion rules need to be written manually, which is costly to maintain and commercial tools do not support logical deletion and DDL synchronization well enough, requiring custom development; on the other hand, trigger-based synchronization solutions intrude into business operations, resulting in significant performance loss, and open-source tools do not implement automatic type conversion and dynamic field mapping.

[0005] Therefore, there is an urgent need for a technical solution that is highly versatile, flexible, and supports real-time synchronization of multi-source heterogeneous databases, especially suitable for large and complex systems such as power grids that have extremely high requirements for data synchronization. Summary of the Invention

[0006] This invention aims to provide a method and system for real-time incremental synchronization of heterogeneous databases for cloud migration, so as to achieve efficient, stable and reliable synchronization of cross-platform database cloud migration, meet the requirements of low latency and high consistency data synchronization, and ensure a smooth transition of business systems.

[0007] This invention solves the above-mentioned technical problems through the following technical solution: a method for real-time incremental synchronization of heterogeneous databases for migration to the cloud, the method comprising the following steps: S1. Capture the change log stream of the local source database in real time using the SourceDump plugin; S2. The change log stream is parsed and reconstructed into a unified event change model through the Transform_UCEF plugin, and redundant information in the change stream is processed through the change stream compression algorithm, recording only the field information that has changed. S3. Use the FieldMap plugin to perform data type conversion, field mapping, add logical deletion markers, and add synchronous timestamp markers. S4. In the Transform_UCEF plugin, respond to DDL type events to achieve table structure synchronization. Only handle DDL events with operation types ALTER TABLE and CREATE TABLE, dynamically map and convert them into DDL statements suitable for the target database, and automatically create or modify the corresponding table structure in the target database. S5. Write change events to the cloud target database and record the synchronization point using the CopyTarget plugin; S6. Package the exception handling and fault tolerance mechanism into a plug-in process. The exception handling and fault tolerance mechanism includes performing full and incremental data verification during the data migration process, and taking corresponding handling measures for data inconsistency, target database unavailability, network and other anomalies that cause transmission interruption. S7. The orchestrated plug-in process is automatically scheduled through the DSServer scheduler.

[0008] On the other hand, the present invention also provides a real-time incremental synchronization system for heterogeneous databases for migration to the cloud, comprising: SourceDump plugin module: Used to capture source database change log streams; The Transform_UCEF plugin module is used to parse logs and reconstruct them into a unified event model, while responding to DDL events to achieve table structure synchronization. The FieldMap plugin module is used to perform data type conversion, field mapping, and tag addition. CopyTarget plugin module: Used to write to the target library and record synchronization points; Exception handling and fault tolerance plugin module: used to perform data validation and exception handling; Plugin orchestration module: Used to provide visual orchestration functionality; DSSERVER scheduling module: used to automate the scheduling of plug-in processes.

[0009] The positive and progressive effects of this invention are as follows: 1. This invention supports high throughput (≥100,000 records / second) and low latency (≤100ms) data synchronization, meeting the high real-time requirements of business operations. It has real-time anomaly handling and breakpoint resume capabilities, with a fault recovery time of less than 1 minute, ensuring the stability and reliability of data synchronization. 2. The system of this invention supports automatic data type conversion and dynamic field mapping between mainstream heterogeneous databases, reducing manual intervention and significantly lowering operation and maintenance costs. It adopts a plug-in and visual orchestration design, which is highly flexible and can be quickly adapted and adjusted according to different business scenarios. It is highly applicable, especially to large and complex systems such as power grids that have extremely high requirements for data synchronization, providing efficient and reliable technical support for their database migration to the cloud. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the system architecture provided by the present invention.

[0011] Figure 2 The unified event change model structure diagram provided by this invention.

[0012] Figure 3 This is a schematic diagram of the exception handling and fault tolerance mechanism provided by the present invention. Detailed Implementation

[0013] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments.

[0014] See Figures 1 to 3 As shown, a real-time incremental synchronization method for heterogeneous databases used for migration to the cloud, taking the migration of data from a local MySQL database to a cloud-based PostgreSQL database as an example, includes the following steps: S1. Capture change log streams: The source databases include MySQL and PostgreSQL, and the change log streams are the binlog logs of MySQL and the wal logs of PostgreSQL, respectively; enable the SourceDump plugin, select MySQL as the source database type in the plugin configuration options, monitor the binlog log generation of the MySQL source database in real time through the SourceDump plugin, and transmit the captured binlog logs to the Transform_UCEF plugin; S2. Parsing and Reconstructing into a Unified Event Model: After receiving the binlog log, the Transform_UCEF plugin parses it, listens to TableMapEvent to obtain the table structure, parses WriteRowsEventData, UpdateRowsEventData, and DeleteRowsEventData to obtain the data changes of insert, update, and delete operations respectively, parses the DDL statements in QueryEvent to extract table structure change information, processes redundant information through a change stream compression algorithm, records only the changed fields, and integrates information such as operation type, table name, field data, primary key, original data value, new data value, changed value, and timestamp according to the unified event change model format, and reconstructs it into a unified event change model; Because the display methods in the change log stream differ across databases, the log parsing process needs to be customized to suit different databases, thus building a unified event change model. For example: MySQL databases establish a binary log connection, specifying the binlog file and location to begin listening; they parse TableMapEvent to obtain table metadata, caching the database name, table name, column definitions, and data types; data change processing includes: insert events: parsing WriteRowsEventData to extract newly added row data; update events: parsing UpdateRowsEventData to extract a comparison of data before and after the change; delete events: parsing DeleteRowsEventData to extract deleted row data; and handling DDL changes by parsing SQL statements in QueryEvent to identify DDL operations such as CREATE / ALTER / DROP, extracting table structure change information, and converting all events into a standard data model. If a PostgreSQL database uses the pgoutput plugin, it can continuously capture database changes by creating logical replication slots and logical decoding slots; poll for WAL changes and periodically extract binary WAL log records from the replication slots; parse binary data according to the pgoutput protocol to identify message types (including Begin / Commit / Relation / Insert / Update / Delete / Truncate); extract DDL operation information, parse table structure changes from Relation messages, including DDL operations such as creating and modifying tables, and obtain complete table structure information, including column definitions, data types, constraints, etc.; and uniformly convert DDL and DML changes into a standardized data model, including operation type, schema, table name, table structure, data changes, etc. For the large amount of redundant information in the change stream, a change stream compression algorithm can be used to process the information in the change stream, recording the changed parts in the change model, clearly recording which fields were specifically modified each time, facilitating status rollback and change history queries. ;

[0015] ;

[0016] ; in, This represents the difference in data state between time t and time t-1. The data state at time t. The data status at time t-1 This represents the set of fields that have changed. `f` is a specific field from all fields in the data model. `id` is the unique identifier for the change event. `op` represents the operation type of this change. `timestamp` is the timestamp of the change. Other metadata.

[0017] As mentioned above, the unified event change model includes DML operation types (CRUD operations) and DDL operation types, timestamps, table names, field information, primary keys, original data values, new data values, and changed values. The method of parsing and reconstructing the change log stream into a unified event change model is encapsulated in the Transform_UCEF plugin. The parsing type is configurable, including DDL events and DML events. This represents the difference in data state between time t and time t-1. The data state at time t. Data status at time t-1 This represents the set of fields that have changed. `f` is a specific field from all fields in the data model. `id` is the unique identifier for the change event. `op` represents the operation type of this change. `timestamp` is the timestamp of the change. Other metadata.

[0018] The change stream compression algorithm further combines the pre-analyzed data distribution characteristics and business rules in the metadata knowledge base; for wide tables that are frequently updated but only a few fields change, the algorithm uses incremental snapshot comparison based on the primary key; for business data with time series or version number characteristics, the algorithm can identify and only synchronize the latest valid version, achieving compression at the business semantic level, further reducing network transmission volume and target database write load.

[0019] S3. Data Processing: The unified event change model is transmitted to the FieldMap plugin. The configurable items of the FieldMap plugin include field mapping rules, data type conversion rules, whether to add logical deletion flags and corresponding fields, and whether to add synchronization timestamp flags and corresponding fields. In the plugin configuration, the data type mapping rules for MySQL and PostgreSQL are preset, the field mapping rules are enabled, and the logical deletion flags and synchronization timestamp flags are enabled. The plugin completes data type conversion and field mapping according to the configuration rules, adds logical deletion flags to events of DELETE operations so that the record can be identified and processed as a logical deletion when output to the target database, and adds the current system time as a synchronization timestamp to each event model for subsequent synchronization time recording and monitoring. Based on the data type mapping relationship between the source and target databases, data type conversion is performed on the data in the unified event change model to ensure that the data can be correctly stored in the target database. Simultaneously, field mapping is performed according to configuration rules, mapping fields in the source database to corresponding fields in the target database to adapt to the target database's table structure. Furthermore, to prevent accidental data deletion, a logical deletion flag can be optionally added to the event change model. When a deletion operation occurs in the source database, the record is marked as logically deleted in the event. A synchronization timestamp can also be optionally added to record the data synchronization time for subsequent monitoring and traceability. The data type conversion and field mapping methods are encapsulated in the FieldMap plugin. Field mapping and its types are configurable, as are the requirements for synchronization timestamps and logical deletions. The FieldMap plugin integrates metadata-driven automatic mapping rule derivation. In the absence of explicit configuration, the plugin can call the mapping knowledge base and automatically generate candidate mapping rules based on field name similarity, data type compatibility, and pattern recognition of sampled data values ​​for user confirmation or direct application, greatly reducing the initial configuration workload for heterogeneous database synchronization.

[0020] S4. Table Structure Synchronization: The Transform_UCEF plugin responds to DDL type events to achieve table structure synchronization. The Transform_UCEF plugin is configured with transformation DDL items, and the plugin will monitor the changes in the table structure of the MySQL source database in real time during runtime. Real-time monitoring of table structure changes in the source database, including the addition, deletion, and modification of fields. When a table structure change is detected in the unified event change model after log parsing, i.e., the operation type is ALTER TABLE or CREATE TABLE, other operations such as DROP TABLE are ignored. Dynamically map and convert to DDL statements suitable for the target database; automatically create or modify the corresponding table structure in the target database to ensure that the table structure of the target database is consistent with that of the source database, so as to ensure that the data can be correctly written to the target database. If it is CREATE TABLE, directly obtain the table structure information in the unified data model, construct the type mapping of the corresponding target table, and form the DDL table creation statement of the target table. If it is ALTER TABLE, it needs to decompose the DDL statement, obtain the specific operation type, table name, comments, etc. in ALTER, construct the type mapping of the corresponding target table, and form the DDL modification statement of the target table. The response method for DDL type events is encapsulated in the Transform_UCEF plugin. The parsing type item is configurable, including DDL events and DML events. When processing DDL events, the Transform_UCEF plugin, in addition to basic table structure mapping, can also handle adaptive transformations of partitioned tables, indexes, and constraints (such as foreign keys and unique constraints) based on a mapping knowledge base. For features specific to cloud-native target databases (such as distributed sharding keys and TTL settings), the plugin can automatically recommend and adapt based on strategy templates. At the same time, for cascading DDL operations involving multiple tables (such as modifying the type of related fields), the plugin can identify the relationships and coordinate the synchronization order of multiple table structures to ensure transaction consistency.

[0021] S5. Writing to the target database and recording the synchronization point: The CopyTarget plugin receives the processed unified event change model, generates insert, update or delete statements according to PostgreSQL SQL syntax, and writes them to the PostgreSQL target database according to the field mapping rules and other configuration items of the FieldMap plugin. During the writing process, the current processing position to the MySQL binlog (such as file name and offset) is recorded as the synchronization point, and the position information is stored in the built-in library. The CopyTarget plugin supports batch commits and transaction control to improve the performance of writing to the cloud target library. The synchronization point is stored in the built-in library for fault recovery and breakpoint resumption after system failure or interruption. The processed unified event change model is written to the target database (such as MySQL, PostgreSQL, Oracle, etc.) according to the target database's write requirements, supporting batch commits and transaction control to improve write performance. After each successful write, a synchronization checkpoint is recorded—the current position synchronized to the source database change log (e.g., binlogposition, LSN)—to facilitate fault recovery and resume interrupted transmission. In the event of a system failure or interruption, synchronization can resume from this checkpoint, ensuring the continuity and reliability of data synchronization. The writing to the cloud target database and the recording of the synchronization checkpoint are encapsulated in the CopyTarget plugin; the target database item is configurable.

[0022] S6. Exception handling and fault tolerance: When the network is interrupted, the local cache component automatically writes the unsynchronized events to the local disk. The size of a single cache file is limited to 1GB and it is stored in fragments according to timestamps. After the network is restored, the system obtains the last successful synchronization point from the built-in library. The SourceDump plugin starts to recapture binlog logs from that point, and subsequent plugins process them according to the process to achieve breakpoint resume. The exception handling and fault tolerance plugin module periodically performs incremental consistency checks, extracting change records from the past hour based on synchronization points. If the verification component detects inconsistencies in the same field value of a record in a table between MySQL and PostgreSQL, it triggers an automatic compensation process: re-pulling the complete data of the record from MySQL, generating a compensation event and marking it as "high priority," synchronizing it to PostgreSQL to overwrite the erroneous value, and logging the repair. The exception handling and fault tolerance mechanism in S6 further includes: intelligent difference detection based on data profiles; the system utilizes the field importance classification stored in the mapping knowledge base (e.g., primary key fields, financial amount fields, and status fields are high-sensitivity fields) to apply different verification frequencies and precisions to fields of different levels during incremental verification; real-time row-level verification is used for high-sensitivity fields; periodic sampling verification can be used for low-sensitivity fields; when inconsistencies are detected, root cause intelligent analysis is performed by combining the change event flow context and field characteristics to distinguish whether it is network packet loss, plugin logic error, or target-side human error, and a differentiated recovery strategy is adopted accordingly to improve fault location and recovery efficiency.

[0023] The exception handling and fault tolerance mechanism is packaged into a plug-in process. This mechanism performs full and incremental data verification during data migration. After initial synchronization, a full consistency check is performed to establish a data baseline between the source and target databases. Simultaneously, a full comparison can be performed periodically during off-peak hours each day using methods such as hash verification. During subsequent incremental migration, incremental consistency checks are performed based on the data baseline and synchronization points, monitoring near-real-time data changes. The entire exception handling and fault tolerance mechanism is encapsulated as a plug-in process, with configurable verification methods. After data migration initialization, a full consistency check is performed to establish a data baseline between the source and target databases; incremental consistency checks are performed based on synchronization points, monitoring near-real-time data changes. Specific exceptions and their handling methods include: In case of data inconsistency: the difference record is located based on the primary key hash value, and compensation synchronization (data is pulled again from the source database) is automatically initiated, and the difference log is recorded to the metadata database; if the inconsistency is caused by a data structure processing error, an alarm will be triggered, requiring manual handling.

[0024] When the target database is unavailable: pause writing and cache change events. Once the target database is restored, resume transmission at the synchronization point. The cache capacity supports peak data volume of ≥2 hours. In cases where network or other anomalies cause transmission interruptions: enable local log caching, automatically verify breakpoints after network recovery, and use incremental synchronization mode to retransmit data during the interruption period.

[0025] S7. Visualized Plugin Orchestration and Automated Scheduling: In the designer, the SourceDump plugin, Transform_UCEF plugin, FieldMap plugin, CopyTarget plugin, and exception handling and fault tolerance plugin are arranged in the above order through visual drag-and-drop. The parameters of each plugin are configured, and the orchestrated plugin process is submitted to the DSServer scheduler. The scheduling cycle is set to real-time triggering. The scheduler automatically monitors changes in the source database and triggers the execution of the plugin process to achieve automated synchronization. Visual orchestration is achieved through plugins, and scheduling automation is achieved through the DSSERVER scheduler. Each of the above requirements must be made into a plugin that allows for process and task orchestration in the designer, with configurable options. Visual drag-and-drop orchestration in the designer enables incremental task configuration according to needs, increasing flexibility. Automated scheduling can be achieved through the DSServer scheduler, reducing human intervention. The DSServer scheduler has load-aware elastic scheduling capabilities; it monitors the processing queue depth, CPU / memory usage, and network I / O of each plugin in real time; when the SourceDump plugin's capture rate is detected to be consistently higher than the processing capacity of subsequent plugins, it can automatically elastically scale up the processing instance for the Transform_UCEF or CopyTarget plugin; conversely, it scales down to save resources. Furthermore, the scheduler can perform predictive scheduling based on historical traffic patterns, pre-scaling before business peaks to ensure synchronization stability.

[0026] S8. Migration Verification: After the data migration initialization is completed, the exception handling and fault tolerance plugin module performs a full consistency check and performs key field hash verification on the core business table to ensure that the hash consistency rate reaches 100% and establishes a data baseline.

[0027] Furthermore, this invention also provides a real-time incremental synchronization system for heterogeneous databases used for migration to the cloud, comprising: SourceDump plugin module: Used to capture source database change log streams; The Transform_UCEF plugin module is used to parse logs and reconstruct them into a unified event model, while responding to DDL events to achieve table structure synchronization. The FieldMap plugin module is used to perform data type conversion, field mapping, and tag addition. CopyTarget plugin module: Used to write to the target library and record synchronization points; Exception handling and fault tolerance plugin module: used to perform data validation and exception handling; Plugin orchestration module: Used to provide visual orchestration functionality; DSSERVER scheduling module: used to automate the scheduling of plug-in processes.

[0028] The SourceDump plugin captures the change log stream of the local source database in real time. It also monitors the binlog generation of the MySQL source database in real time and transmits the captured binlog logs to the Transform_UCEF plugin. For the PostgreSQL target database, log capture is not required on the target side; the focus is on capturing logs from the source MySQL database.

[0029] Example 1:

[0030] The Transform_UCEF plugin parses and reconstructs the change log stream into a unified event change model. Within the Transform_UCEF plugin, the MySQL binlog is parsed, extracting information such as the operation type (e.g., INSERT, UPDATE, DELETE, including DDL operations like ALTER TABLE), table name, field list, and corresponding data values ​​for each change record. Then, it is formatted according to the preset unified event change model format, such as... Figure 2 As shown, this information is reconstructed into a unified event model that includes operation type, table name, field data, logical deletion flag (initially not deleted), and synchronization timestamp (current time). Implement data type conversion and field mapping using the FieldMap plugin. See the following items in the FieldMap plugin in sequence.

[0031] Data type conversion: Since there are differences between the data types of MySQL and PostgreSQL, for example, the BIGINT type in MySQL corresponds to the INT8 type in PostgreSQL. In the data processing module, the data types of MySQL are converted to the corresponding data types of PostgreSQL according to the preset data type mapping table. Field mapping: If the table field names in the source database MySQL are inconsistent with the table field names in the target database PostgreSQL, the field mapping configuration will map the source field names to the target field names, ensuring that the data can be correctly written to the corresponding fields in the target table. Logical deletion flag: When a DELETE operation is parsed in the binlog, a logical deletion flag can be added so that the target database can recognize and process the record as a logical deletion.

[0032] Synchronization timestamp: Add the current system time as a synchronization timestamp to each event model for subsequent synchronization time recording and monitoring. The Transform_UCEF plugin responds to DDL type events to achieve table structure synchronization. Configure transformation DDL items in the Transform_UCEF plugin. During runtime, the plugin monitors changes to the table structure of the MySQL source database in real time. For example, when a new field "name" is added to the MySQL table `table`, the model after log parsing detects an operation type of `ALTER TABLE`. It identifies and decomposes the entire operation statement, extracting the specific change type as `ADD COLUMN`, with the table name `table`, the field name `name`, and the field type as `varchar(255)`. A corresponding type mapping is constructed for the PostgreSQL target table, also as `varchar(255)`. This is integrated to form a new table structure modification statement, adding the corresponding "name" field to the target table.

[0033] The CopyTarget plugin writes change events to the cloud target database and records the synchronization point. The CopyTarget plugin generates insert, update, or delete statements from the unified event model processed by the Transform_UCEF plugin according to PostgreSQL SQL syntax, and writes them to the PostgreSQL target database based on the field mapping rules of the FieldMap plugin and other configuration items. During the writing process, it records the current processing position in the MySQL binlog (such as filename and offset) as the synchronization point and stores this point information in the built-in database.

[0034] Exception handling and fault tolerance mechanisms.

[0035] Example 2: In the event of a network interruption, the local caching component automatically writes the unsynchronized events to the local disk, with a single file size limit of 1GB and fragmented by timestamp. After the network is restored, the system obtains the last successful point from the synchronization point management module and executes the steps starting from (1) to retransmit sequentially.

[0036] Example 3: In the event of data inconsistency, the verification component detects that the value of the same field in a record in a table is inconsistent between MySQL and PostgreSQL, triggering an automatic compensation process: the complete data of the record is pulled from MySQL, a compensation event is generated and marked as "high priority", the error value is synchronized to PostgreSQL, and the log is recorded after the repair.

[0037] Migration verification Full verification: After initial synchronization is complete, perform hash verification on key fields of the core business table (hash consistency rate 100%).

[0038] Incremental verification: Every hour, based on the synchronization point, extract the change records of the "previous hour", compare the final state of the source end and the target end, and ensure that the inconsistency rate is ≤0.001%.

[0039] The above data migration method is implemented through a plugin to achieve visual drag-and-drop orchestration. The verification of exception handling and fault tolerance mechanism is encapsulated into one of the plugin processes. The scheduling is automated through the DSSERVER scheduler. The scheduler can be executed on a regular basis or according to the conditions in the verification process, reducing human intervention.

[0040] This invention is not limited to the embodiments described above. Any changes in shape or structure shall fall within the protection scope of this invention. The protection scope of this invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of this invention, but all such changes and modifications shall fall within the protection scope of this invention.

Claims

1. A method for real-time incremental synchronization of heterogeneous databases for migration to the cloud, characterized in that, The method includes the following steps: S1. Capture the change log stream of the local source database in real time using the SourceDump plugin; S2. The change log stream is parsed and reconstructed into a unified event change model through the Transform_UCEF plugin, and redundant information in the change stream is processed through the change stream compression algorithm, recording only the field information that has changed. S3. Use the FieldMap plugin to perform data type conversion, field mapping, add logical deletion markers, and add synchronous timestamp markers. S4. In the Transform_UCEF plugin, respond to DDL type events to achieve table structure synchronization. Only handle DDL events with operation types ALTER TABLE and CREATE TABLE, dynamically map and convert them into DDL statements suitable for the target database, and automatically create or modify the corresponding table structure in the target database. S5. Write change events to the cloud target database and record the synchronization point using the CopyTarget plugin; S6. Package the exception handling and fault tolerance mechanism into a plug-in process. The exception handling and fault tolerance mechanism includes performing full and incremental data verification during the data migration process, and taking corresponding handling measures for data inconsistency, target database unavailability, network and other anomalies that cause transmission interruption. S7. The orchestrated plug-in process is automatically scheduled through the DSServer scheduler.

2. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that, The source database in S1 includes at least MySQL and PostgreSQL, with corresponding change log streams of MySQL's binlog log and PostgreSQL's wal log, respectively.

3. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that, The unified event change model in S2 includes DML operation type, DDL operation type, timestamp, table name, field information, primary key, original data value, new data value, and changed value; The specific process of parsing and reconstructing the change log stream in S2 is as follows: For MySQL databases, connect to the MySQL replication stream, listen to TableMapEvent to obtain table structure, parse WriteRows EventData, UpdateRows EventData and DeleteRows EventData to obtain data changes, parse DDL statements in QueryEvent, extract table name, operation type, column definition and data changes, and convert them into a unified model; For PostgreSQL databases, changes are obtained by creating logical replication slots. The binary data is parsed according to the protocol followed by the logical decoding output plugin, the message type is identified, and table structure changes, including DDL operations for creating and modifying tables, are parsed from the Relation messages. The DDL and DML changes are then uniformly converted into a standardized data model.

4. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that, The formula for processing change information in the change stream compression algorithm in S2 is as follows: ； ； ； in, This represents the difference in data state between time t and time t-1. The data state at time t. The data status at time t-1 This represents the set of fields that have changed. `f` is a specific field from all fields in the data model. `id` is the unique identifier for the change event. `op` represents the operation type of this change. `timestamp` is the timestamp of the change. Other metadata.

5. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 2, characterized in that: The configurable items of the FieldMap plugin in S3 include field mapping rules, data type conversion rules, whether to add logical deletion markers and corresponding fields, and whether to add synchronization timestamp markers and corresponding fields.

6. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that: In S4: When handling the CREATE TABLE event, directly obtain the table structure information in the unified data model, construct the type mapping of the corresponding target table, and form the DDL table creation statement of the target table; When handling the ALTER TABLE event, the DDL statement is decomposed to obtain the specific operation type, table name, and comment information in ALTER, construct the type mapping of the corresponding target table, and form the DDL modification statement of the target table.

7. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 6, characterized in that: The target database in S5 includes at least MySQL, PostgreSQL and Oracle, and the synchronization point is the current synchronization position to the source database change log, including the binlog position of MySQL and the LSN of PostgreSQL; The CopyTarget plugin in S5 supports batch commits and transaction control. Synchronization points are stored in a built-in library for fault recovery and resume transmission after system failure or interruption.

8. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that: In the case of data inconsistency, S6 locates the difference record based on the primary key hash value, automatically initiates compensation synchronization to pull the data from the source database, and records the difference log to the metadata database. If the inconsistency is caused by a data processing structure error, an alarm is triggered and manual handling is required. The anomaly handling and fault tolerance mechanism in S6 further includes: intelligent difference detection based on data profiles; the system utilizes the field importance classification stored in the mapping knowledge base to apply different verification frequencies and precisions to fields of different levels during incremental verification; real-time row-level verification is used for highly sensitive fields; timed sampling verification is used for low-sensitivity fields; when inconsistency is detected, root cause intelligent analysis is performed by combining the change event flow context and field characteristics to distinguish whether it is network packet loss, plugin logic error or target end human error, and a differentiated recovery strategy is adopted accordingly to improve fault location and recovery efficiency.

9. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that: In S6, when the target database is unavailable, writing is paused and change events are cached. The cache capacity supports peak data volume of ≥2 hours. Once the target database is restored, the transmission will resume at the synchronization point. In response to transmission interruptions caused by network or other anomalies, local log caching is enabled. Once the network is restored, the breakpoint is automatically verified, and incremental synchronization mode is used to retransmit the data during the interruption.

10. The method for real-time incremental synchronization of heterogeneous databases for cloud migration as described in claim 1, characterized in that: The S7 designer's visual drag-and-drop arrangement function supports flexible adjustment of the plugin combination order and plugin configuration items according to different business needs, so as to adapt to the incremental synchronization requirements of heterogeneous databases in different scenarios.

11. The system for real-time incremental synchronization of heterogeneous databases for migration to the cloud as described in any one of claims 1-10, characterized in that: include: SourceDump plugin module: Used to capture source database change log streams; The Transform_UCEF plugin module is used to parse logs and reconstruct them into a unified event model, while responding to DDL events to synchronize table structures. The FieldMap plugin module is used to perform data type conversion, field mapping, and tag addition. CopyTarget plugin module: Used to write to the target library and record synchronization points; Exception handling and fault tolerance plugin module: used to perform data validation and exception handling; Plugin orchestration module: Used to provide visual orchestration functionality; DSSERVER scheduling module: used to automate the scheduling of plug-in processes.

Images