A method and system for three-way data synchronization between heterogeneous systems

Abstract

This invention relates to the field of data integration technology, specifically providing a method and apparatus for three-party data synchronization between heterogeneous systems. First, the heterogeneous data layer includes multiple independent participants, each system maintaining its own independent data model and technology stack. The adaptation and access layer deploys data synchronization clients and metadata adapters on each participant, enabling bidirectional conversion between local models and standard intermediate models. The central coordination layer is a central synchronization server. The asynchronous message layer distributes asynchronous instructions based on message queues and topics. The unified service layer provides synchronization monitoring, log auditing, and rule configuration. Compared with existing technologies, this invention can meet the high reliability and loose coupling requirements of multi-party data collaboration in scenarios such as smart cities and unified government services.

Description

Technical Field

[0001] This invention relates to the field of data integration technology, specifically providing a method and system for three-way data synchronization between heterogeneous systems. Background Technology

[0002] The complexity of point-to-point interface integration increases exponentially when multiple systems are synchronized (N*(N-1) interfaces need to be developed for N systems), leading to tight inter-system dependencies and maintenance difficulties. ETL methods based on centralized databases typically involve scheduled batch synchronization, resulting in data latency of up to several hours, which cannot meet near real-time business requirements. The lack of a unified data model standard makes it difficult to adapt to differences in data models across different technical architectures (such as Oracle / MySQL / WebService). The lack of effective global version management and conflict detection mechanisms easily leads to data inconsistency during concurrent updates.

[0003] Traditional Enterprise Service Bus (ESB) configurations are cumbersome and inflexible, with poor support for capturing and synchronizing incremental data changes in real time. In the event of network jitter or system failures, the lack of end-to-end acknowledgment and retry mechanisms makes data synchronization easily interrupted and difficult to trace. Centralized databases can easily become performance bottlenecks, making it difficult to support data synchronization across large-scale system clusters. Summary of the Invention

[0004] This invention addresses the shortcomings of the prior art by providing a highly practical method for three-way data synchronization between heterogeneous systems.

[0005] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable three-party data synchronization system for heterogeneous systems.

[0006] The technical solution adopted by this invention to solve its technical problem is: A three-party data synchronization method for heterogeneous systems: First, the heterogeneous data layer contains multiple independent participants, and each system maintains its own independent data model and technology stack. The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model; The central coordination layer is the central synchronization server; The asynchronous messaging layer dispatches asynchronous instructions based on topics using message queues. The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

[0007] Furthermore, the metadata adapter encapsulates mapping rules, performs complex field type conversions and semantic alignment, so that key information and business semantics are not lost when data is converted between standard and local formats.

[0008] Furthermore, in the central coordination layer, a globally unique and incremental version identifier is assigned for each data change, and multiple resolution strategies are preset. When updating data, the target party verifies the continuity of the version number.

[0009] Furthermore, production and consumption are decoupled based on message queues. After the central server issues a command, it waits for the target party to confirm the receipt. If no confirmation is received within the timeout period, a retry or alarm is triggered. Based on the message offset and checkpoint mechanism, data is not lost or duplicated after fault recovery.

[0010] A three-party data synchronization system for heterogeneous systems, firstly, the heterogeneous data layer contains multiple independent participants, each system maintaining its own independent data model and technology stack; The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model; The central coordination layer is the central synchronization server; The asynchronous messaging layer dispatches asynchronous instructions based on topics using message queues. The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

[0011] Furthermore, the metadata adapter encapsulates mapping rules, performs complex field type conversions and semantic alignment, so that key information and business semantics are not lost when data is converted between standard and local formats.

[0012] Furthermore, in the central coordination layer, a globally unique and incremental version identifier is assigned for each data change, and multiple resolution strategies are preset. When updating data, the target party verifies the continuity of the version number.

[0013] Furthermore, production and consumption are decoupled based on message queues. After the central server issues a command, it waits for the target party to confirm the receipt. If no confirmation is received within the timeout period, a retry or alarm is triggered. Based on the message offset and checkpoint mechanism, data is not lost or duplicated after fault recovery.

[0014] Compared with existing technologies, the method and system for three-way data synchronization between heterogeneous systems proposed in this invention have the following significant advantages: This invention employs a star topology of "centralized coordination + asynchronous messaging," completely replacing complex point-to-point integration and significantly reducing system coupling and operational complexity. Through a standard intermediate model and adapter, it effectively solves the "language barrier" problem between heterogeneous systems, enabling data interoperability regardless of the technology stack. It integrates a triple mechanism of global version management, multi-strategy conflict resolution, and end-to-end confirmation to ensure eventual data consistency in complex network and concurrent environments.

[0015] The central server features a stateless design, and the message queue is easily horizontally scalable, supporting the addition of more participants and handling higher concurrent synchronous traffic.

[0016] The full-link operation log provides complete audit clues for data synchronization, greatly improving system observability and troubleshooting efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an architecture diagram of a three-way data synchronization method for heterogeneous systems. Figure 2 This is a schematic diagram of a standard model mapping in a three-way data synchronization method for heterogeneous systems. Figure 3 This is a flowchart of conflict detection and resolution in a three-way data synchronization method used between heterogeneous systems; Figure 4 It is a reliable event transmission sequence diagram used in a three-way data synchronization method between heterogeneous systems; Figure 5 This is a schematic diagram of the operation log auditing interface in a three-party data synchronization method used between heterogeneous systems. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The following is a preferred embodiment: Example

[0021] like Figure 1 As shown in this embodiment, a three-party data synchronization method for heterogeneous systems is firstly implemented by a heterogeneous data layer containing multiple independent participants (such as a population database Oracle, a housing database MySQL, and an enterprise database API), with each system maintaining its own independent data model and technology stack. The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model (based on JSON Schema); The central coordination layer is the central synchronization server; The asynchronous messaging layer distributes asynchronous commands by topic based on message queues (such as Kafka / RocketMQ); The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

[0022] like Figure 2 As shown, the metadata adapter encapsulates mapping rules, supports complex field type conversions and semantic alignment, and ensures that key information and business semantics are not lost when data is converted between standard and local formats. It also supports hot updates of adapter rules to adapt to changes in business models.

[0023] like Figure 3 As shown, a globally unique and incremental version ID is assigned to each data change. Multiple solutions are preset, including "most recent update priority" based on timestamps, "source system priority" based on business rules, or manual intervention. When updating data, the target party verifies the continuity of the version number to ensure the sequential nature and eventual consistency of data changes.

[0024] like Figure 4 As shown, message queues are used to decouple production and consumption, mitigating sudden traffic spikes. After the central server issues a command, it waits for confirmation from the target; if no confirmation is received within a timeout period, a retry or alarm is triggered. Based on message offset and checkpoint mechanisms, data is ensured to be neither lost nor duplicated after fault recovery. Example

[0025] Smart city population-housing-enterprise data synchronization: (1) Initialization configuration: Define a standard JSON Schema model to describe people, houses, and enterprises. Develop and deploy metadata adapters for the public security population database (Oracle), housing and construction database (MySQL), and industrial and commercial enterprise database (Web Service).

[0026] (2) Change capture and publication: When citizen Wang’s registered address changes, the population database client captures this update through log parsing, converts it into a standard JSON event (including global version v6) through the adapter, and publishes it to the central server.

[0027] (3) Central processing and routing: The central server verifies the data format and queries the current version (v5) of the record in the target system (house database, enterprise database). After the conflict detection passes, the rule engine determines that this change only needs to be routed to the house database.

[0028] (4) Asynchronous instruction distribution: Generate a synchronization instruction for the housing database and send it to the "Population-Housing Synchronization" topic through a message queue.

[0029] (5) Localized update: The housing database consumption command converts standard data into MySQL UPDATE statements through its adapter, and sends confirmation to the central server after the data update is completed.

[0030] (6) Audit traceability: The entire chain of synchronous events (content, status, version, confirmation information) is persisted to the operation log library for audit query. Example

[0031] A three-party data synchronization system for heterogeneous systems, firstly, the heterogeneous data layer contains multiple independent participants, each system maintaining its own independent data model and technology stack; The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model; The central coordination layer is the central synchronization server; The asynchronous messaging layer dispatches asynchronous instructions based on topics using message queues. The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

[0032] The metadata adapter encapsulates mapping rules, performs complex field type conversions and semantic alignment, and ensures that key information and business semantics are not lost when data is converted between standard and local formats.

[0033] In the central coordination layer, a globally unique and incremental version identifier is assigned for each data change, and multiple resolution strategies are preset. When updating data, the target party verifies the continuity of the version number.

[0034] Decoupling of production and consumption is achieved through message queues. After the central server issues a command, it waits for the target party to confirm the response. If no confirmation is received within a timeout period, a retry or alarm is triggered. Based on the message offset and checkpoint mechanism, data is not lost or duplicated after fault recovery.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for three-way data synchronization between heterogeneous systems, characterized in that, First, the heterogeneous data layer includes multiple independent participants, and each system maintains its own independent data model and technology stack; The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model; The central coordination layer is the central synchronization server; The asynchronous messaging layer dispatches asynchronous instructions based on topics using message queues. The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

2. The method for tripartite data synchronization between heterogeneous systems according to claim 1, characterized in that, The metadata adapter encapsulates mapping rules, performs complex field type conversions and semantic alignment, and ensures that key information and business semantics are not lost when data is converted between standard and local formats.

3. The method for three-way data synchronization between heterogeneous systems according to claim 2, characterized in that, In the central coordination layer, a globally unique and incremental version identifier is assigned for each data change, and multiple resolution strategies are preset. When updating data, the target party verifies the continuity of the version number.

4. The method for three-way data synchronization between heterogeneous systems according to claim 3, characterized in that, Decoupling of production and consumption is achieved through message queues. After the central server issues a command, it waits for the target party to confirm the response. If no confirmation is received within a timeout period, a retry or alarm is triggered. Based on the message offset and checkpoint mechanism, data is not lost or duplicated after fault recovery.

5. A three-way data synchronization system for heterogeneous systems, characterized in that, First, the heterogeneous data layer includes multiple independent participants, and each system maintains its own independent data model and technology stack; The adaptation and access layer deploys data synchronization clients and metadata adapters on each participating party to achieve bidirectional conversion between the local model and the standard intermediate model; The central coordination layer is the central synchronization server; The asynchronous messaging layer dispatches asynchronous instructions based on topics using message queues. The unified service layer provides synchronous monitoring, log auditing, and rule configuration.

6. A three-way data synchronization system for heterogeneous systems according to claim 5, characterized in that, The metadata adapter encapsulates mapping rules, performs complex field type conversions and semantic alignment, and ensures that key information and business semantics are not lost when data is converted between standard and local formats.

7. A three-way data synchronization system for heterogeneous systems according to claim 6, characterized in that, In the central coordination layer, a globally unique and incremental version identifier is assigned for each data change, and multiple resolution strategies are preset. When updating data, the target party verifies the continuity of the version number.

8. A three-way data synchronization system for heterogeneous systems according to claim 7, characterized in that, Decoupling of production and consumption is achieved through message queues. After the central server issues a command, it waits for the target party to confirm the response. If no confirmation is received within a timeout period, a retry or alarm is triggered. Based on the message offset and checkpoint mechanism, data is not lost or duplicated after fault recovery.

Images