A cross-data-center data synchronization apparatus

Abstract

This invention provides a cross-data center data synchronization device, relating to the fields of distributed data processing and network communication technology. The device integrates three core modes: offline batch synchronization, real-time incremental synchronization, and message queue integration. Through a plug-in design, it is compatible with various heterogeneous data sources, including databases, file storage, and message queues. It incorporates an intelligent network adaptation unit, achieving stable transmission in weak network environments through network quality quantification detection, dynamic transmission strategy adjustment, and breakpoint resumption mechanisms. It supports customized processing such as SQL filtering, field mapping, and data replacement, as well as cross-center bidirectional real-time synchronization, while providing unified monitoring and audit logs. This invention achieves "one system covering all synchronization needs," with synchronization latency ≤100ms, a task success rate ≥99.9% in weak network environments, and 100% heterogeneous data source adaptation rate. It significantly reduces the deployment costs of multiple tools for enterprises and ensures the consistency, real-time performance, and reliability of data across data centers.

Description

Technical Field

[0001] This invention relates to the field of distributed data processing and network communication technology, specifically to a cross-data center data synchronization device. Background Technology

[0002] As enterprises deepen their digital transformation, data sharing and collaboration across data centers and business systems have become core requirements. Data synchronization, as a key link in distributed data management, directly affects business continuity and the realization of data value.

[0003] However, existing cross-data center data synchronization technologies have many unresolved pain points:

[0004] Insufficient adaptability to multiple scenarios necessitates the deployment of multiple tools: Enterprises simultaneously have three types of needs: offline batch data migration (such as daily full backup), real-time business data synchronization (such as transaction data), and message flow integration (such as log messages). However, existing technologies are mostly single-mode tools (such as traditional ETL which only supports offline, and single real-time tools which do not support message integration), which leads to enterprises needing to deploy multiple independent systems, increasing operation and maintenance costs and data consistency risks.

[0005] Unstable transmission in weak network environments: Cross-data center networks suffer from problems such as latency, bandwidth fluctuations, packet loss, and even interruptions. Traditional solutions lack dynamic adaptation mechanisms, and batch synchronization needs to be re-run after an interruption. Real-time synchronization is prone to data backlog or loss, and message synchronization is prone to packet loss and duplication.

[0006] Poor compatibility with heterogeneous data sources: Enterprises have diverse data source types (databases, files, message queues) and different brands (MySQL, Oracle, MinIO, Kafka, etc.). Existing solutions mostly only support a single type or a few types of data sources, which cannot meet the synchronization needs of complex heterogeneous environments.

[0007] Weak customization capabilities: Different business scenarios have personalized requirements for data synchronization (such as filtering invalid data, field structure conversion, and compatibility with target end configuration). Traditional solutions rely on complex script development, which lacks flexibility and makes it difficult to quickly adapt to business changes.

[0008] The operation and maintenance is difficult and lacks unified monitoring: multiple tools are operated and maintained independently, without a unified task management, status monitoring and problem tracing mechanism. Data consistency verification and anomaly investigation require manual operation, which is extremely inefficient.

[0009] Existing technologies cannot achieve "full-scenario coverage, stability in weak networks, heterogeneous compatibility, flexible customization, and convenient operation and maintenance" of collaboration. There is an urgent need for an integrated cross-data center data synchronization solution to fill the technological gap in the industry. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention provides a cross-data center data synchronization device. It addresses the core pain points of existing cross-data center data synchronization technologies, such as insufficient adaptability to multiple scenarios, unstable transmission in weak network environments, poor compatibility with heterogeneous data sources, weak customization capabilities, and high operation and maintenance costs. The invention innovatively constructs an integrated architecture of "EventMesh triggering hub + dual integrated modules + intelligent network adaptation".

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A cross-data center data synchronization device, the system comprising a data source end, a data / message integration platform, a data target end, and an EventMesh triggering center;

[0013] The data source provides data to be synchronized, including multiple types of heterogeneous data sources, specifically:

[0014] Databases: MySQL, Oracle, openGauss, TiDB;

[0015] File storage class: MinIO;

[0016] Message queue class: Kafka;

[0017] The data / message integration platform is the core processing layer of the tool, which is divided into two sub-modules: data integration and message integration. Each sub-module contains a network adaptation unit and a corresponding business processing unit.

[0018] The data target end receives synchronized data. The receiving end is heterogeneous and compatible with the source system, including database, file storage, and message queue types corresponding to the source system.

[0019] The EventMesh triggering hub is deployed between the source system and the integration platform, and supports three triggering methods:

[0020] Scheduled synchronization: Triggers batch / offline data synchronization;

[0021] Real-time synchronization: Triggers real-time synchronization of incremental data;

[0022] Consume messages: Triggers message synchronization in the message queue.

[0023] Furthermore, the data / message integration platform adopts a plug-in architecture, with Reader / Writer / Source / Sink components supporting custom development and scalable adaptation to new types of data sources; the unified monitoring module provides a visual display of task status, synchronization rate, network quality, and data consistency, and the audit log records the entire process operation, supporting abnormal alarms and problem tracing.

[0024] Furthermore, the synchronization device is implemented through a data synchronization method, including the following steps:

[0025] Step 1: Trigger Mode Selection

[0026] The EventMesh trigger hub sends "scheduled synchronization", "real-time synchronization" or "consume messages" trigger commands to the data / message integration platform according to business needs, corresponding to offline batch synchronization, real-time incremental synchronization and message queue integration scenarios, respectively.

[0027] Step 2: Intelligent Network Adaptation

[0028] The network adaptation unit of the data / message integration platform detects the network status across data centers through quantitative indicators and dynamically adjusts the transmission strategy (such as incremental / full reading, transmission rate, fragment size, etc.).

[0029] Step 3: Data / Message Access and Processing

[0030] The data integration module reads data from the database / file storage according to the trigger command, and processes it through filtering, replacement, and field mapping; the message integration module consumes message queue data, and processes it through aggregation and format unification.

[0031] Step 4: Data / Message Writing and Synchronization

[0032] The processed data stream / message stream is written to the corresponding storage / queue on the target end to achieve cross-data center synchronization; after a weak network interruption, synchronization is resumed from the breakpoint to avoid data loss or duplication;

[0033] Step 5: Monitoring and Auditing

[0034] Real-time monitoring of synchronization task status, network quality, and data consistency; generation of audit logs; and support for anomaly alerts and problem tracing.

[0035] Furthermore, both the data source and the target end mentioned in step 1 support heterogeneous adaptation, including database types (MySQL, Oracle, openGauss, TiDB), file storage types (MinIO), and message queue types (Kafka, RabbitMQ, RocketMQ). Flexible expansion and access to the data source are achieved through pluggable Reader / Writer components.

[0036] Furthermore, the quantitative detection indicators of the network adaptation unit in step 2 include network latency (ping value), bandwidth utilization (iperf3), and packet loss rate, and the network quality is divided into 1-10 levels (level 1 is the best); the transmission strategy adjustment rules are as follows: when the network quality is ≤ level 5, it automatically switches to incremental reading + fragmented transmission (batch synchronization), reduces the data capture rate (real-time synchronization), and enables message caching + slowed consumption (message synchronization).

[0037] Furthermore, the specific process of offline batch synchronization described in step 3 is as follows: EventMesh initiates a timed command → the network adaptation unit determines the transmission mode → the data access unit reads the full / incremental data → the data processing unit executes "filtering (selecting valid data according to business rules) → replacement (compatible with incompatible content on the target end) → field mapping (structure conversion)" → the data writing unit writes the data to the target end in batches.

[0038] Furthermore, the real-time incremental synchronization described in step 3 is achieved through the Change Data Capture CDC (CDC) technology, specifically: the network adaptation unit dynamically adjusts the capture rate → the change real-time capture unit captures data source addition, deletion, and modification operations → the data processing unit performs customized processing → the data writing unit writes to the target end in real time, with a synchronization delay of ≤100ms.

[0039] Furthermore, the specific process of message integration in step 3 is as follows: EventMesh initiates a message consumption command → the network adaptation unit adjusts the consumption rate → the message access unit consumes the source end message queue data → the message aggregation unit classifies and merges multi-topic messages → the message monitoring unit detects the transmission status → the message writing unit writes to the target end message queue, supporting message deduplication and packet loss compensation.

[0040] Furthermore, the breakpoint resume mechanism described in step 4 is implemented by recording the synchronization progress: batch synchronization records the data read offset, real-time synchronization records the CDC position, and message synchronization records the consumption offset; after the network is restored, synchronization continues from the recorded progress to avoid a full rerun.

[0041] Furthermore, the system supports bidirectional real-time synchronization across data centers. By configuring dual change data capture tasks, incremental changes at the data source and target ends are monitored separately, synchronized with each other, and data conflicts are handled through a conflict detection algorithm to ensure bidirectional data consistency.

[0042] This invention provides a cross-data center data synchronization device. It has the following beneficial effects:

[0043] 1. This invention provides a cross-data center data synchronization device, which adopts an architecture of "EventMesh triggering hub + data integration + message integration". It integrates three modes: offline batch, real-time incremental and message integration. One system covers all synchronization needs across data centers, completely replacing the deployment of multiple tools. It proposes a three-in-one weak network adaptation solution of "quantitative detection - dynamic adjustment - breakpoint resume". Through network quality classification of 1-10 levels, the transmission strategy is dynamically optimized. Breakpoint resume avoids full rerun, and the success rate of weak network tasks is increased to 99.9%.

[0044] 2. This invention provides a cross-data center data synchronization device. Its Reader / Writer / Source / Sink components adopt a plug-in design, support custom development, and can quickly adapt to new types of data sources. It is currently compatible with mainstream data sources such as MySQL, Oracle, openGauss, TiDB, MinIO, and Kafka, with a 100% compatibility rate. It achieves cross-center bidirectional synchronization through dual CDC task configuration and innovates a conflict detection algorithm based on "timestamp + version number" to automatically handle concurrent modification conflicts and ensure bidirectional data consistency.

[0045] 3. This invention provides a cross-data center data synchronization device with a built-in visual rule configuration interface, supporting customized operations such as SQL filtering, field mapping, data replacement, and format conversion. No code development is required, and it can quickly adapt to different business scenarios. Based on the primary key / timestamp, large batch synchronization tasks are split into multiple independent sub-tasks, supporting concurrent execution. If a single sub-task fails, it can be retried independently, improving synchronization efficiency by more than 50%.

[0046] 4. This invention provides a cross-data center data synchronization device that provides visualized monitoring of task status, synchronization rate, network quality, and data consistency. The audit log records the entire process operation, trigger commands, synchronization progress, and abnormal information, and supports abnormal alarms and rapid tracing. Attached Figure Description

[0047] Figure 1 This is a diagram showing the overall architecture of the cross-data center data synchronization device of the present invention.

[0048] Figure 2 This is a flowchart of the data integration module of the present invention;

[0049] Figure 3 This is a flowchart of the message integration module of the present invention;

[0050] Figure 4 This is a flowchart of the bidirectional real-time synchronization conflict handling process of the present invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0052] In the description of this invention, unless otherwise explicitly specified and limited, the terms "connected," "linked," and "fixed" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0053] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0054] In the description of this embodiment, the terms "upper," "lower," "right," etc., refer to the orientation or positional relationship shown in the accompanying drawings. They are used only for ease of description and simplification of operation, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first" and "second" are used only for distinction in description and have no special meaning.

[0055] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0056] like Figures 1-4 As shown, this embodiment of the invention provides a cross-data center data synchronization device, the core architecture of which is "data source end → EventMesh triggering hub → data / message integration platform → data target end";

[0057] The data source provides data to be synchronized, including multiple types of heterogeneous data sources, specifically:

[0058] Databases: MySQL, Oracle, openGauss, TiDB;

[0059] File storage class: MinIO;

[0060] Message queue class: Kafka.

[0061] The data / message integration platform is the core processing layer of the tool, which is divided into two sub-modules: data integration and message integration. Each sub-module contains a network adaptation unit and a corresponding business processing unit.

[0062] The core components of the data / message integration platform are the data integration module, the message integration module, the network adaptation unit, and the unified monitoring module;

[0063] Data integration module:

[0064] The scheduled synchronization submodule reads full / incremental data according to the trigger command, and after processing through "data filtering (filtering according to business rules, such as status=1) → data replacement (such as IP address, storage path replacement) → field mapping (structure transformation, such as source_col→target_col)", it is written to the target end in batches.

[0065] Real-time synchronization submodule: Captures data source addition, deletion and modification operations through CDC technology, and writes them to the target end in real time after customized processing consistent with timed synchronization, with a synchronization delay of ≤100ms.

[0066] Message integration module:

[0067] Message ingestion: Multi-topic messages from the source message queue (Kafka / RabbitMQ, etc.);

[0068] Message aggregation: Classify and merge messages from multiple sources, and unify data formats (such as JSON standardization);

[0069] Message monitoring: Detects message transmission status and performs packet loss compensation and deduplication;

[0070] Message writing: Write the processed message to the target message queue.

[0071] Network adaptation unit:

[0072] Quantitative testing: Using ping (latency) and iperf3 (bandwidth / packet loss rate) tools, network quality is divided into 1-10 levels (level 1 is the best, level 10 is the worst).

[0073] Strategy adjustment: When the network quality is ≤5 (weak network), batch synchronization is switched to incremental reading + fragmented transmission (fragment size adjustable from 1-10MB), real-time synchronization reduces the CDC capture rate (adjustable from 100-1000 messages / second), and message synchronization slows down consumption + enables local caching;

[0074] Resume interrupted download: Records synchronization progress, batch synchronizes and records offsets, real-time synchronizes and records CDC positions, and records message offsets. Resumes from the breakpoint after network recovery.

[0075] Unified monitoring module:

[0076] Visual monitoring: Displays task status (running / paused / failed), synchronization rate, network quality, and data consistency;

[0077] Audit logs record information throughout the entire process, including trigger commands, synchronization progress, exception information, and operators.

[0078] Anomaly alerts: Notify users of situations such as task failure, network anomalies, and data inconsistencies via SMS / email / platform pop-up.

[0079] The data target end receives synchronized data. The receiving end is heterogeneous and compatible with the source system, including database, file storage, and message queue types corresponding to the source system.

[0080] The EventMesh trigger hub is deployed between the source system and the integration platform, supporting three triggering methods:

[0081] Scheduled synchronization: Triggers offline batch synchronization at a preset period (e.g., 2 AM daily), suitable for full data migration and batch backup;

[0082] Real-time synchronization: Continuously monitors changes in the data source and triggers real-time incremental synchronization, suitable for transaction data and business status data;

[0083] Consume messages: Real-time triggering of message queue data consumption and synchronization, suitable for log messages and event notification data.

[0084] This cross-datacenter data synchronization method includes the following steps:

[0085] Step 1: Trigger Mode Selection and Configuration

[0086] Business personnel can configure synchronization tasks through a visual interface: select the trigger mode (such as "real-time synchronization"), data source type (such as MySQL), target type (such as openGauss), and customized rules (such as filtering invalid state data and field mapping relationships);

[0087] The EventMesh trigger hub sends corresponding trigger commands according to the configuration.

[0088] Step 2: Intelligent Network Adaptation

[0089] The network adaptation unit starts detection, obtains network latency (e.g., 30ms), bandwidth utilization (e.g., 40%), and packet loss rate (e.g., 0.1%), and determines the network quality to be level 3 (excellent). The real-time synchronization submodule captures data at the default rate (500 data entries / second).

[0090] If the network deteriorates to level 6 (weak network) later, the capture rate will be automatically adjusted to 200 records / second.

[0091] Step 3: Data Access and Processing

[0092] The real-time synchronization submodule captures incremental changes in transaction data in the MySQL database using CDC technology, filters out invalid data with status=0 according to configuration rules, replaces the storage path in the data (e.g., replacing " / data / center1 / " with " / data / center2 / "), and maps the "trade_no" field to the "order_no" field on the target end.

[0093] Step 4: Data writing and breakpoint resumption

[0094] The processed transaction data is written to the openGauss database in real time.

[0095] If the network is suddenly interrupted, the system records the current CDC position (e.g., binlog file + offset).

[0096] Once the network is restored, data will continue to be captured from that location to avoid data loss or duplicate writing.

[0097] Step 5: Monitoring and Auditing

[0098] The unified monitoring module displays the synchronization rate (e.g., 450 messages / second), synchronization latency (e.g., 80ms), and network quality (level 3) in real time. The audit log records the task start time, triggering command, amount of data processed, and no abnormal information.

[0099] If data inconsistency occurs, an alarm will be triggered immediately and details of the conflicting data will be recorded.

[0100] Implementation Case:

[0101] Taking the cross-regional data synchronization of a large internet company as an example, this company has two data centers in Beijing and Shanghai, and needs to achieve:

[0102] Scenario 1: Daily full user data synchronization from the Beijing center's MySQL database to the Shanghai center's openGauss database, offline batch synchronization;

[0103] Scenario 2: Real-time transaction logs from the Kafka queue in the Beijing center are synchronized to the RabbitMQ queue in the Shanghai center for message integration;

[0104] Scenario 3: Real-time update files from the Shanghai center's MinIO are synchronized to the Beijing center's MinIO, with real-time incremental synchronization;

[0105] Network environment: The cross-center network has occasional latency of 10-50ms and bandwidth fluctuations, which is a weak network scenario.

[0106] System Configuration:

[0107] Data source: Beijing center (MySQL 8.0, Kafka 2.8), Shanghai center (MinIO 2023);

[0108] Target platforms: Shanghai Center (openGauss 3.0, RabbitMQ 3.10), Beijing Center (MinIO 2023);

[0109] EventMesh trigger hub: Configure scheduled synchronization (daily at 2 AM), real-time synchronization (continuously triggered), and message consumption (continuously triggered);

[0110] Network adaptation unit: detection cycle 10s, fragment size default 5MB, real-time synchronization rate default 500 records / second;

[0111] Customized rules: When MySQL → openGauss synchronization, filter invalid user data with status=0, and map the field "user_id" to "uid"; when Kafka → RabbitMQ synchronization, use JSON as the unified message format.

[0112] The implementation steps are as follows:

[0113] Scenario 1 (Offline Batch Synchronization):

[0114] Every day at 2 AM, EventMesh sends a "timed synchronization" command;

[0115] The network adaptation unit detected network quality as level 4 (good), and selected full read mode;

[0116] The data integration module reads all user data from the Beijing center's MySQL database, filters invalid data, and completes field mapping.

[0117] Batch writing to the Shanghai Center openGauss database, synchronizing 10 million data entries, took 40 minutes, with no data loss.

[0118] Scenario 2 (Message Integration):

[0119] EventMesh sends a "consume message" command, which is continuously triggered;

[0120] The network adaptation unit detects network quality fluctuations down to level 6 (weak network), automatically reduces the consumption rate to 200 messages / second, and enables local caching.

[0121] The message integration module consumes transaction logs from the Beijing center's Kafka, and aggregates them into a unified JSON format.

[0122] The data was written to the RabbitMQ queue at the Shanghai Center. During this period, the network was interrupted for 5 minutes. After the network was restored, the synchronization continued from the cached offset without any packet loss or duplication.

[0123] Scenario 3 (Real-time Incremental Synchronization):

[0124] EventMesh sends a "real-time synchronization" command, which is continuously triggered;

[0125] The network adaptation unit detected network quality as Level 3 (Excellent) and captured incremental file changes from Shanghai Center MinIO at the default rate.

[0126] The data integration module handles file path replacement (e.g., replacing " / data / sh / " with " / data / bj / ").

[0127] Real-time writing to the Beijing center using MinIO, with an average synchronization latency of 60ms and 100% file consistency.

[0128] Implementation results:

[0129] Full-scenario coverage: One system completes three types of synchronization needs, eliminating the need to deploy multiple tools and reducing operation and maintenance costs by 75%;

[0130] Weak network adaptation: In scenario 2, there is no data loss after network interruption, and the success rate of weak network tasks is 100%.

[0131] Synchronization performance: Offline synchronization rate of 250,000 messages / minute, real-time synchronization latency ≤100ms, message synchronization rate of 200-500 messages / second;

[0132] Data consistency: No data loss, duplication or inconsistency occurred in any of the three scenarios, and the consistency verification accuracy was 100%.

[0133] The following points should be noted in this article:

[0134] 1. The accompanying drawings of the embodiments disclosed herein only relate to the structures involved in the embodiments disclosed herein; other structures can be referred to in a general design.

[0135] 2. Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0136] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A cross-data center data synchronization device, characterized in that, This includes the data source, data / message integration platform, data target, and EventMesh triggering hub; The data source provides data to be synchronized, including multiple types of heterogeneous data sources, specifically: Databases: MySQL, Oracle, openGauss, TiDB; File storage class: MinIO; Message queue class: Kafka; The data / message integration platform is the core processing layer of the tool, which is divided into two sub-modules: data integration and message integration. Each sub-module contains a network adaptation unit and a corresponding business processing unit. The data target end receives synchronized data. The receiving end is heterogeneous and compatible with the source system, including database, file storage, and message queue types corresponding to the source system. The EventMesh triggering hub is deployed between the source system and the integration platform, and supports three triggering methods: Scheduled synchronization: Triggers batch / offline data synchronization; Real-time synchronization: Triggers real-time synchronization of incremental data; Consume messages: Triggers message synchronization in the message queue.

2. The cross-data center data synchronization device according to claim 1, characterized in that, The data / message integration platform adopts a plug-in architecture. The Reader / Writer / Source / Sink components support custom development and can be extended to adapt to new types of data sources. The unified monitoring module provides a visual display of task status, synchronization rate, network quality, and data consistency. The audit log records the entire process and supports abnormal alarms and problem tracing.

3. The cross-data center data synchronization device according to claim 1, characterized in that, The synchronization device is implemented through a data synchronization method, including the following steps: Step 1: Trigger Mode Selection The EventMesh triggering hub sends "scheduled synchronization", "real-time synchronization" or "consume messages" trigger commands to the data / message integration platform according to business needs, corresponding to offline batch synchronization, real-time incremental synchronization and message queue integration scenarios, respectively. Step 2: Intelligent Network Adaptation The network adaptation unit of the data / message integration platform detects the cross-data center network status through quantitative indicators and dynamically adjusts the transmission strategy. Step 3: Data / Message Access and Processing The data integration module reads data from the database / file storage according to the trigger command, and processes it through filtering, replacement, and field mapping; the message integration module consumes message queue data, and processes it through aggregation and format unification. Step 4: Data / Message Writing and Synchronization The processed data stream / message stream is written to the corresponding storage / queue on the target end to achieve cross-data center synchronization; after a weak network interruption, synchronization is resumed from the breakpoint to avoid data loss or duplication; Step 5: Monitoring and Auditing Real-time monitoring of synchronization task status, network quality, and data consistency; generation of audit logs; and support for anomaly alerts and problem tracing.

4. The cross-data center data synchronization device according to claim 3, characterized in that, Both the data source and the target end mentioned in step 1 support heterogeneous adaptation, including database, file storage, and message queue types. Flexible expansion and access of the data source are achieved through pluggable Reader / Writer components.

5. The cross-data center data synchronization device according to claim 3, characterized in that, The quantitative detection indicators of the network adaptation unit in step 2 include network latency, bandwidth utilization, and packet loss rate, and the network quality is divided into 1-10 levels; the transmission strategy adjustment rules are as follows: when the network quality is ≤5, it automatically switches to incremental reading + fragmented transmission, reduces the data capture rate, and enables message caching + slowed consumption.

6. The cross-data center data synchronization device according to claim 3, characterized in that, The specific process of offline batch synchronization described in step 3 is as follows: EventMesh initiates a timed command → the network adaptation unit determines the transmission mode → the data access unit reads the full / incremental data → the data processing unit executes "filtering → replacement → field mapping" → the data writing unit writes the data to the target end in batches.

7. The cross-data center data synchronization device according to claim 3, characterized in that, The real-time incremental synchronization described in step 3 is achieved through the Change Data Capture CDC (CDC) technology, specifically: the network adaptation unit dynamically adjusts the capture rate → the real-time capture unit changes the data source for adding, deleting, and modifying operations → the data processing unit performs customized processing → the data writing unit writes the data to the target end in real time, with a synchronization delay of ≤100ms.

8. The cross-data center data synchronization device according to claim 3, characterized in that, The specific process of message integration in step 3 is as follows: EventMesh initiates a message consumption command → Network adaptation unit adjusts the consumption rate → Message access unit consumes the source end message queue data → Message aggregation unit classifies and merges multi-topic messages → Message monitoring unit detects the transmission status → Message writing unit writes to the target end message queue, supporting message deduplication and packet loss compensation.

9. The cross-data center data synchronization device according to claim 3, characterized in that, The breakpoint resume mechanism described in step 4 is achieved by recording the synchronization progress: batch synchronization records the data read offset, real-time synchronization records the CDC position, and message synchronization records the consumption offset; after the network is restored, synchronization continues from the recorded progress to avoid a full rerun.

10. The cross-data center data synchronization device according to claim 1, characterized in that, The system supports bidirectional real-time synchronization across data centers. By configuring dual change data capture tasks, it monitors incremental changes at the data source and target ends respectively, synchronizes with each other, and handles data conflicts through a conflict detection algorithm to ensure bidirectional data consistency.

Images