Real-time fusion and incremental synchronization method and system for heterogeneous data sources

Abstract

The application discloses a real-time fusion and incremental synchronization method and system of heterogeneous data sources, and belongs to the technical field of data synchronization. The method comprises the following steps: a change data capture step is used to parse a plurality of database logs through a source connector to obtain change events; an event standardization step is used to convert the change events into a unified format; a stream processing step is used for deduplication, sorting and transaction boundary identification; an adaptive scheduling step is used to dynamically adjust the consumption parallelism according to the load; and a consistency checking step is used to compare the data digest of the source end and the target end and repair inconsistencies. The application solves the problems of high synchronization delay, complex integration and difficult consistency guarantee of the heterogeneous data sources, controls the end-to-end delay in the order of seconds, and supports the processing of millions of events per second.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, specifically to a method and system for real-time fusion and incremental synchronization of heterogeneous data sources. Background Technology

[0002] As enterprises deepen their digital transformation, the data generated by various business systems is growing exponentially, and the data storage formats are becoming increasingly diversified. Enterprises typically run multiple types of database systems simultaneously, including relational databases such as MySQL, PostgreSQL, and Oracle, as well as document-oriented databases such as MongoDB and message brokers such as Kafka. The need for data synchronization between these heterogeneous data sources is becoming increasingly urgent, serving as a critical infrastructure to support real-time data analysis, business decision-making, and cross-system collaboration.

[0003] Existing technologies offer various data synchronization solutions. This invention patent CN118503915A discloses a method and system for multi-source heterogeneous data fusion. This solution achieves heterogeneous data fusion processing by constructing a unified data model and a policy generation model. Specifically, the solution collects historical heterogeneous data from different data sources for preprocessing, constructs a policy generation model based on a deep Q-network algorithm to automatically generate data mapping and transformation policies, uses the RF-BiLSTM-Attention algorithm to construct a homogeneous data fusion model for feature extraction and fusion, and finally uses the IPCO-DBN algorithm to construct a fused data analysis model to analyze and predict the fusion results. This solution achieves certain results at the data fusion level, eliminating the heterogeneity of different data sources and solving the data silo problem.

[0004] However, the existing technologies described above have the following technical shortcomings in real-time incremental synchronization scenarios. First, synchronization latency is difficult to control at the second level. Existing solutions are mainly designed for batch data fusion and lack mechanisms for capturing and transmitting real-time changed data, failing to meet the low-latency requirements of real-time analysis services. Second, different databases have significantly different change capture mechanisms. MySQL uses binary logs, PostgreSQL uses write-ahead logs, and Oracle uses redo logs. Existing solutions do not provide a unified change data capture framework, resulting in high integration complexity and high development and maintenance costs. Third, data consistency during incremental synchronization is difficult to guarantee, especially when cross-source transactions are involved. Due to the different commit times of transactions from different data sources, intermediate states of data inconsistency can easily occur. Fourth, existing solutions do not consider the dynamic fluctuations in the change rate of data sources. When a data source experiences a sudden surge in changes, the fixed processing capacity can lead to data backlog and a sharp increase in synchronization latency.

[0005] To address the aforementioned technical issues, there is an urgent need for a data synchronization method and system that can capture changes from multiple heterogeneous data sources in real time, uniformly convert them into a standardized format, ensure cross-source transaction consistency, and dynamically adjust processing capabilities according to load. Summary of the Invention

[0006] To address the technical problems of high real-time synchronization latency of heterogeneous data sources, complex integration of change capture mechanisms, and difficulty in guaranteeing cross-source data consistency in existing technologies, this invention provides a method and system for real-time fusion and incremental synchronization of heterogeneous data sources.

[0007] The present invention provides a method for real-time fusion and incremental synchronization of heterogeneous data sources, comprising a change data capture step, an event standardization step, a stream processing step, an adaptive scheduling step, and a consistency verification step. The change data capture step connects to multiple heterogeneous data sources through a source-end connector, parses database transaction logs or message queues to obtain data change events. Heterogeneous data sources include relational databases and message middleware. Data change events include operation type, primary key identifier, value before change, value after change, and transaction identifier. The event standardization step converts data change events into a standardized message format. The standardized message format includes a source identifier field, a table identifier field, an operation type field, a primary key field, a data payload field, a transaction identifier field, and a timestamp field, generating standardized change messages. The stream processing step performs deduplication and temporal sorting on the standardized change messages, identifies transaction boundaries based on the transaction identifier field, aggregates standardized change messages belonging to the same transaction into transaction message groups, and outputs an ordered sequence of transaction message groups. The adaptive scheduling step collects the change rate of each data source and the write latency of the target end. When the number of pending transaction message groups exceeds a preset data backlog threshold, the number of consumer threads is increased; when the synchronization latency is lower than a preset latency recovery threshold, the number of consumer threads is decreased. The adjusted consumption parallelism is applied to the stream processing step. The consistency verification step periodically collects the source and target data digests, compares the differences between the source and target data digests, generates a data consistency verification report, and triggers an incremental repair process when data inconsistency is detected.

[0008] Preferably, relational databases include MySQL, PostgreSQL, and Oracle, and message middleware includes Kafka and RabbitMQ.

[0009] Preferably, the data backlog threshold ranges from 1,000 to 100,000 transaction message groups, and the delay recovery threshold ranges from 100 milliseconds to 5,000 milliseconds.

[0010] Preferably, the verification period for consistency checks ranges from 1 minute to 60 minutes.

[0011] This invention also provides a real-time fusion and incremental synchronization system for heterogeneous data sources, including a change data capture module, an event standardization module, a stream processing module, an adaptive scheduling module, and a consistency verification module. The change data capture module connects to multiple heterogeneous data sources through a source-end connector and parses database transaction logs or message queues to obtain data change events. The event standardization module converts data change events into a standardized message format, generating standardized change messages. The stream processing module performs deduplication and temporal sorting on the standardized change messages, identifies transaction boundaries based on transaction identifiers, and outputs an ordered sequence of transaction message groups. The adaptive scheduling module collects the change rate of each data source and the write latency of the target end, dynamically adjusting the consumption parallelism based on data backlog and synchronization latency. The consistency verification module periodically collects source-end data summaries and target-end data summaries, compares differences, and generates a data consistency verification report.

[0012] The beneficial effects of this invention are as follows: By constructing a unified change data capture framework, it supports real-time change capture of various mainstream databases and message middleware, reducing the complexity of integrating heterogeneous data sources; by standardizing event processing, it converts change events of different formats into a unified format, simplifying downstream processing logic; by using transaction boundary identification and aggregation mechanisms, it ensures the atomicity of cross-source transactions, solving the data consistency problem; by using an adaptive parallelism adjustment mechanism, it achieves dynamic matching between processing capacity and load, automatically scaling up during peak periods to ensure low latency, and automatically scaling down during off-peak periods to reduce resource consumption; end-to-end synchronization latency is controlled at the second level, supporting the processing of millions of change events per second, meeting the needs of real-time analysis business. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method for real-time fusion and incremental synchronization of heterogeneous data sources provided in the embodiments of the present invention.

[0014] Figure 2 This is an architecture diagram of the heterogeneous data source real-time fusion and incremental synchronization system provided in the embodiments of the present invention. Detailed Implementation

[0015] Please refer to the attached document. Figures 1-2 The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0016] like Figure 1 As shown in the figure, this invention provides a method for real-time fusion and incremental synchronization of heterogeneous data sources. This method achieves real-time incremental data synchronization between multiple heterogeneous data sources through the deep collaboration of five core steps: change data capture, event standardization, stream processing, adaptive scheduling, and consistency verification. The specific implementation of each step will be described in detail below.

[0017] The change data capture step is the starting point of the entire synchronization process, responsible for acquiring data change events from various heterogeneous data sources in real time. In one embodiment of the invention, this step achieves interfaceing with different types of data sources by deploying multiple source-end connectors.

[0018] For MySQL databases, the source connector connects to the master node by simulating a MySQL slave node and reads the binary log to obtain data change records. The binary log records all data modification operations in the form of events, including INSERT, UPDATE, and DELETE operations. The connector parses the ROW format events of the binary log to extract the database name, table name, operation type, primary key value of the affected row, and complete row data before and after the change. In a preferred embodiment of the invention, the connector maintains a log position record to identify the currently parsed binary log file name and offset position, ensuring that reading can continue from the breakpoint after the connector restarts, avoiding data loss or duplication.

[0019] For PostgreSQL databases, the source connector connects to the database server via a logical replication protocol and reads the logically decoded output of the Write-Ahead Log (WAL). PostgreSQL's logical decoding function converts physical WAL records into logical change events. The connector uses the pgoutput plugin as the output plugin to obtain change events containing complete row data. To ensure reliable transmission, the connector uses a replication slot mechanism. The PostgreSQL server retains all WAL segments after the corresponding position in the replication slot, ensuring that changed data is not lost even if the connector temporarily goes offline.

[0020] For Oracle databases, the source connector obtains change data through the LogMiner tool or by directly parsing the redo log file. In a preferred embodiment of the invention, raw log parsing is used instead of LogMiner to achieve higher collection performance. The raw log parser directly reads the binary content of online redo logs and archived log files, extracting data change information by analyzing the log block structure and change vector records. Compared to the LogMiner method, raw log parsing can achieve an event collection rate of over 20,000 records per second, with less impact on the performance of the source database.

[0021] For MongoDB databases, the source connector connects to the replica set and monitors data changes in real time via the Change Stream API. The Change Stream is implemented based on MongoDB's oplog mechanism, capturing collection-level INSERT, UPDATE, REPLACE, and DELETE operations and returning change details in BSON document format. The connector uses a Resume Token to record read positions, supporting read recovery from any point in time.

[0022] For the Kafka message middleware, the source connector acts as a consumer, subscribing to a specified topic and reading message records as change events. The connector maintains a consumption offset for each partition, supporting consumption from a specified offset to ensure no message loss. In embodiments of this invention, the connector supports configuring consumer group IDs to achieve load balancing and failover across multiple connector instances.

[0023] The data change event output by the change data capture step includes the following core fields: Operation type field identifies the change type, with values ​​including INSERT, UPDATE, and DELETE; Primary key identifier field records the primary key value of the changed record, and for composite primary keys, the values ​​of each primary key column are stored in an ordered list; Before change value field stores the complete row data before the UPDATE and DELETE operations; After change value field stores the complete row data after the INSERT and UPDATE operations; Transaction identifier field records the unique identifier of the transaction to which the change belongs, used for subsequent transaction boundary identification.

[0024] The event standardization step receives the raw change events output from the change data capture step and converts them into a unified, standardized message format. Since change event formats from different data sources vary significantly, standardization is a crucial step in achieving unified management of heterogeneous data sources.

[0025] In embodiments of this invention, the standardized message format adopts a JSON structure and includes the following fields: The source identifier field (sourceId) records a unique identifier for the data source, using a four-segment naming convention of "type:host:port:database name", such as "mysql:192.168.1.100:3306:order_db", facilitating downstream processing modules to identify the data source. The table identifier field (tableId) records a complete table identifier in the form of "schema name.table name" for relational databases, "database name.collection name" for MongoDB, and the topic name for Kafka. The operation type field (operation) uniformly uses single-character encoding: INSERT operation is encoded as "I", UPDATE operation as "U", DELETE operation as "D", transaction start as "B", and transaction commit as "C". The primary key field (primaryKey) stores the mapping relationship between primary key column names and corresponding values ​​in the form of a JSON object. The payload field (payload) includes two subfields: before value and after value, storing the name and value of each column in key-value pairs. The transaction identifier field (transactionId) stores the transaction identifier generated by the original data source. For MySQL, this is GTID or XID; for PostgreSQL, it is the transaction ID (xid); and for Oracle, it is the system change number (SCN). The timestamp field (timestamp) records the time when the change event occurred, stored in Unix millisecond timestamp format.

[0026] During the standardization process, it is necessary to handle the data type differences between different data sources. In a preferred embodiment of the present invention, a data type mapping rule table is established to uniformly map the native types of each data source to a set of standard types. Numeric types are uniformly converted to integer (INTEGER), long integer (BIGINT), or floating-point type (DECIMAL), with the DECIMAL type retaining its original precision and decimal places. Time types are uniformly converted to ISO 8601 format string representation, including time zone information to avoid time ambiguity in cross-time zone scenarios. Binary types are uniformly converted to strings using Base64 encoding for storage. Special types, such as MySQL's ENUM, PostgreSQL's array types, and MongoDB's nested documents, are all converted to a JSON-compatible representation.

[0027] The event standardization process also supplements metadata information. In embodiments of the present invention, the following metadata is added to each standardized message: the message sequence number is an auto-incrementing integer used to ensure the global order of the message; the partition key is calculated based on the source identifier and the table identifier and is used for partition routing of the message; and the message version identifies the version number of the standardized message format, supporting forward compatibility evolution of the format.

[0028] The stream processing step is the core processing step of this invention. It is responsible for deduplicating, sorting, and identifying transaction boundaries of standardized change messages to ensure the orderliness of change events and the atomicity of transactions.

[0029] The deduplication sub-step is used to identify and filter duplicate change messages. In a distributed environment, due to network jitter or connector restarts, the same change may be sent multiple times; therefore, deduplication is a necessary step to ensure data integrity. In embodiments of this invention, a hash-based deduplication mechanism is employed. For each standardized change message, the hash value of its deduplication key is calculated. The deduplication key consists of a combination of the source identifier field, table identifier field, primary key field, and transaction identifier field. The system maintains a set of hash values ​​within a time window. When the hash value of a new message already exists in the set, it is determined to be a duplicate message and discarded. In a preferred embodiment of this invention, the deduplication window size is set to 10 minutes, which is sufficient to cover duplicate scenarios caused by network latency and retries. The hash value set is implemented using a Bloom filter, achieving a balance between memory usage and query efficiency, supporting deduplication judgment of millions of messages per second.

[0030] The timing-based sorting sub-step ensures that change messages are processed in the order they occurred. Since clocks from different data sources may deviate, and messages may become out of order during transmission, global sorting is necessary. In this embodiment of the invention, an event-time-based sorting strategy is employed. The system determines the event time based on the timestamp field of the standardized message and uses a watermark mechanism to handle delayed messages. The watermark represents the point in time when the system believes it will not receive any messages with an earlier timestamp, calculated as follows:

[0031] ,

[0032] in, for The water level value at that moment, This is the current processing time. In the embodiments of the present invention, the preset maximum allowable delay time is... The value ranges from 1 second to 30 seconds. When the message delay of a certain data source exceeds... If the message is late, the system will record the late message log and decide whether to discard it or bypass it based on the configuration.

[0033] The transaction boundary identification sub-step is a key mechanism to ensure the atomicity of cross-source transactions. In the embodiments of this invention, a multi-source transaction boundary identification algorithm is designed to identify the start and end positions of a transaction by parsing the transaction identifier and transaction control message. When the stream processing module receives a standardized change message of operation type "B" (transaction start), it creates a new transaction message group object, recording the transaction identifier and start timestamp. Subsequent change messages with the same transaction identifier are added to this transaction message group in sequence, organized into an ordered list according to the order of receipt. When a message of operation type "C" (transaction commit) is received, the corresponding transaction message group is closed, its status is marked as committable, and statistical information such as the number of change messages contained in the transaction and the number of tables involved are calculated.

[0034] In a preferred embodiment of the invention, the aggregation process of transaction message groups takes into account transaction timeouts and exception handling. The system sets a timeout timer for each incomplete transaction message group, with the timeout value ranging from 30 to 300 seconds. If a transaction message group does not receive a commit message within the timeout period, it is determined that the transaction is interrupted, and the system discards all change messages in that transaction message group and records an alarm log. This mechanism avoids the infinite accumulation of transaction message groups due to data source failures or network interruptions.

[0035] The output of the stream processing step is an ordered sequence of transaction message groups, each containing complete transaction change data, which can be used as an atomic unit for subsequent processing. In embodiments of the present invention, the data structure of the transaction message group includes: a transaction identifier (transactionId), a transaction start timestamp (beginTimestamp), a transaction commit timestamp (commitTimestamp), a list of change messages (changes), and transaction metadata (metadata). The messages in the change message list are arranged in the execution order within the original transaction to ensure correctness during replay.

[0036] The adaptive scheduling step achieves dynamic matching between processing capacity and load, representing an innovative design in this invention to improve system resilience and resource utilization. This step dynamically adjusts the consumption parallelism of the stream processing steps based on real-time monitored metrics, optimizing resource allocation while ensuring synchronization latency.

[0037] In embodiments of the present invention, the adaptive scheduling step collects the following monitoring metrics: Change Rate: Records the number of change events generated per second by each data source, obtained through counting and statistics in the change data capture step. Write Latency: Records the write response time of the target database, obtained by calculating the time difference between request initiation and response return after write completion. Backlog Count: Records the number of currently pending transaction message groups, obtained by querying the internal queue depth of the stream processing step. End-to-End Latency: Records the total latency from change occurrence to synchronization completion, calculated by comparing the transaction commit timestamp and synchronization completion timestamp.

[0038] Based on the above monitoring indicators, this invention proposes an adaptive parallelism dynamic adjustment algorithm, the core calculation formula of which is as follows:

[0039] ,

[0040] in, For the adjusted parallelism of consumption, Given the current level of parallel consumption, This represents the current backlog quantity. For data backlog threshold, For the current end-to-end delay, The target latency threshold, This is the backlog sensitivity coefficient. This is the delay sensitivity coefficient. In an embodiment of the present invention, The value range is from 0.3 to 0.8. The value ranges from 0.2 to 0.6, and the specific value is adjusted according to the latency sensitivity of the business scenario.

[0041] In a preferred embodiment of the present invention, the parallelism adjustment follows the following constraints: the range of parallelism values ​​is limited to a preset minimum value. and maximum value To avoid excessive resource release or over-occupancy, adjustments should be limited to no more than 50% of the current value to prevent drastic fluctuations from affecting system stability. Adjustments should be spaced at least 30 seconds apart to avoid the overhead of frequent adjustments. In a preferred embodiment of the invention, a capacity expansion trigger coefficient is set to determine whether the number of consumer threads needs to be increased. This coefficient is defined as the ratio of the number of pending transaction message groups to the data backlog threshold, ranging from 1.5 to 3, preferably 2. When the ratio exceeds the capacity expansion trigger coefficient, the number of consumer threads increases to 1.25 to 1.5 times the current value. When the synchronization delay is lower than the delay recovery threshold for 2 to 5 consecutive sampling periods, the number of consumer threads decreases to 0.5 to 0.8 times the current value.

[0042] The execution logic of the adaptive scheduling step is as follows: The system collects monitoring indicators at fixed intervals (preferably 10 seconds). When the backlog exceeds the data backlog threshold, the system is determined to be under high load. A new parallelism is calculated according to the formula above and adjusted upwards. During adjustment, the system sends a parallelism adjustment command to the stream processing step, which dynamically creates new consumer threads and allocates pending transaction message groups. When the end-to-end latency is detected to be lower than the latency recovery threshold for three consecutive sampling periods, the system load is determined to have returned to normal. A new parallelism is calculated according to the formula and adjusted downwards. During downward adjustment, the system stops some consumer threads and reclaims related resources.

[0043] In an embodiment of the present invention, the data backlog threshold The value ranges from 1,000 to 100,000 transaction message groups, and the delay recovery threshold is... The value range is from 100 milliseconds to 5000 milliseconds. Preferably, the data backlog threshold is set to 5000 records, the delay recovery threshold is set to 500 milliseconds, and the target delay threshold is... Set to 200 milliseconds. With this configuration, the system can quickly scale up during bursts of traffic, keeping synchronization latency below 1 second; and scale down promptly after traffic subsides, reducing resource usage to below 50% of the baseline level.

[0044] The consistency check step is the last line of defense to ensure the correctness of data synchronization. By periodically comparing the data at the source and target ends, it detects and repairs potential data inconsistencies.

[0045] In embodiments of the present invention, consistency verification employs an incremental verification strategy to avoid the performance overhead of full comparison. The system records the time point and verification range of each verification, and subsequent verifications only compare the subset of data that has changed since the last verification.

[0046] The data digest calculation employs an incremental verification algorithm based on a Merkle tree. In this embodiment, the data of each table is divided into multiple data blocks according to the primary key range, with each data block ranging from 1000 to 10000 rows in size. For each data block, the hash values ​​of all its rows are calculated and organized into a Merkle tree structure. The leaf nodes of the Merkle tree store the hash values ​​of individual rows, the non-leaf nodes store the combined hash values ​​of their child nodes, and the hash value of the root node is the digest of the entire data block. The formula for calculating the data block digest is as follows:

[0047] ,

[0048] in, For data block digests, For the first The hash value of the row data. The number of rows contained in the data block. This represents the bitwise XOR operation of the hash value. For the hash function, the SHA-256 algorithm is preferred.

[0049] The consistency comparison process first compares the root node digests of corresponding data blocks on the source and target sides. If the root node digests are the same, the data blocks are considered consistent, and no further comparison is needed. If the root node digests are different, the child nodes are recursively compared to quickly locate the specific inconsistent rows. The time complexity of this mechanism is O(n log n). Compared to line-by-line comparison The complexity has been greatly reduced.

[0050] When data inconsistency is detected, an incremental repair process is triggered. In an embodiment of the present invention, the repair process includes the following sub-steps: First, locate the primary key range of the inconsistent data and record the list of primary key values ​​for the rows that need to be repaired; then, read the complete row data from the source end according to the primary key values; finally, overwrite the read data into the corresponding position on the target end and update the verification status to repaired. To avoid conflicts between the repair process and normal synchronization, the system pauses incremental synchronization of the corresponding table during the repair period and resumes it after the repair is completed.

[0051] In a preferred embodiment of the present invention, the consistency verification period ranges from 1 minute to 60 minutes, preferably set to 5 minutes. A verification period that is too short will increase the system load, while a period that is too long may delay the detection of inconsistency issues. The data consistency verification report generated by the system includes: the verification time range, the number of tables and data blocks verified, the number of consistent data blocks, the number of inconsistent data blocks, a list of primary keys for inconsistent rows, and repair operation records. The verification report supports output in JSON format, facilitating integration into monitoring and alarm systems.

[0052] The consistency verification step and the change data capture step form a closed-loop feedback. When multiple consecutive verifications find that the same data source has a high inconsistency rate, the system automatically adjusts the capture parameters of the data source, such as reducing the log reading batch size and increasing the number of retries, thereby reducing the probability of data loss from the source.

[0053] In the embodiments of the present invention, the performance of the above method was verified through performance testing. The test environment included: one source database each of MySQL 8.0, PostgreSQL 14, and MongoDB 6.0, configured with an 8-core CPU, 32GB of memory, and 500GB of SSD storage; one target database of PostgreSQL 14, with the same configuration; and four intermediate processing nodes, configured with a 16-core CPU and 64GB of memory.

[0054] The test dataset contains 100 tables across three data sources, totaling 500GB of data. The test scenario simulates real-world business change patterns, with the change rate gradually increasing from 1000 changes per second to 1 million changes per second. Test results show that under a steady-state load of 100,000 change events per second, the average end-to-end synchronization latency is 320 milliseconds, and the 99th percentile latency is 890 milliseconds. When the load surges to 1 million change events per second, the adaptive scheduling mechanism expands the parallelism from 8 to 32 within 15 seconds, with a peak synchronization latency of 2.1 seconds. After the load decreases, the parallelism recovers to 12 within 45 seconds. Consistency checks complete the verification of all data blocks within a 5-minute period, with a detected inconsistency rate of less than 0.001%.

[0055] Compared to the scheme in the prior art document CN118503915A, this invention offers significant improvements in real-time performance, integration complexity, and resource utilization. The scheme in the prior art document requires pre-training of the strategy generation and fusion models, resulting in lengthy initialization times and an inability to adapt to dynamic changes in the data source. In contrast, this invention employs a real-time capture mechanism based on log parsing, allowing for immediate deployment without pre-training and achieving dynamic response to load changes through adaptive scheduling.

[0056] like Figure 2 As shown, this embodiment of the invention also provides a real-time fusion and incremental synchronization system for heterogeneous data sources. The system includes a change data capture module 1, an event standardization module 2, a stream processing module 3, an adaptive scheduling module 4, and a consistency verification module 5.

[0057] The change data capture module 1 is used to connect to various heterogeneous data sources through source connectors and parse database transaction logs or message queues to obtain data change events. In embodiments of the present invention, the change data capture module 1 includes a connector management unit, a log parsing unit, and an event output unit. The connector management unit is responsible for maintaining the connection status with each data source and supports connection establishment, disconnection and reconnection, and health checks. The log parsing unit parses different types of database log formats, such as MySQL binary log parsing, PostgreSQL write-ahead log parsing, and Oracle redo log parsing, as described in the aforementioned method embodiments. The event output unit sends the parsed change events to the event standardization module 2.

[0058] Event standardization module 2 is used to convert the data change events output by change data capture module 1 into a standardized message format, generating standardized change messages. In embodiments of the present invention, event standardization module 2 includes a format conversion unit and a metadata supplementation unit. The format conversion unit converts various heterogeneous change events into a unified JSON structure according to the standardized message format defined in the aforementioned method embodiments. The metadata supplementation unit adds additional information such as message sequence number and partition key to each message.

[0059] Stream processing module 3 is used to deduplicate and sort standardized change messages according to their temporal order, identify transaction boundaries based on transaction identifiers, and output an ordered sequence of transaction message groups. In embodiments of the present invention, stream processing module 3 includes a deduplication unit, a sorting unit, and a transaction aggregation unit. The deduplication unit employs the Bloom filter-based deduplication mechanism described in the preceding method embodiments. The sorting unit employs a watermark-based event-time sorting strategy. The transaction aggregation unit implements transaction boundary identification and aggregation of transaction message groups. Stream processing module 3 receives parallelism adjustment instructions from adaptive scheduling module 4 and dynamically adjusts the number of consumer threads.

[0060] The adaptive scheduling module 4 is used to collect the change rate of each data source and the write latency of the target end, and dynamically adjust the consumption parallelism based on the data backlog and synchronization latency. In this embodiment of the invention, the adaptive scheduling module 4 includes an indicator collection unit, a scheduling decision unit, and an instruction issuance unit. The indicator collection unit obtains the change rate indicator from the change data capture module 1, the write latency indicator from the target end, and the backlog quantity indicator from the stream processing module 3. The scheduling decision unit calculates the optimal parallelism according to the adaptive parallelism dynamic adjustment algorithm in the aforementioned method embodiment. The instruction issuance unit sends the parallelism adjustment instruction to the stream processing module 3.

[0061] The consistency verification module 5 is used to periodically collect source and target data summaries, compare differences, and generate a data consistency verification report. In this embodiment, the consistency verification module 5 includes a summary calculation unit, a comparison unit, and a repair unit. The summary calculation unit uses the Merkle tree-based incremental verification algorithm described in the aforementioned method embodiments to calculate the data summary. The comparison unit compares the source and target summaries to locate inconsistent data blocks and rows. The repair unit executes an incremental repair process to resynchronize inconsistent data from the source to the target. The consistency verification module 5 feeds back the verification results to the change data capture module 1 for optimizing capture parameters.

[0062] In the system embodiment of the present invention, the data flow relationship between the modules is as follows: the output of the change data capture module 1 is sent to the event standardization module 2, the output of the event standardization module 2 is sent to the stream processing module 3, the output of the stream processing module 3 is sent to the target database and notifies the consistency verification module 5; the adaptive scheduling module 4 collects monitoring indicators from the change data capture module 1, the stream processing module 3, and the target database, and issues scheduling instructions to the stream processing module 3; the consistency verification module 5 collects data summaries from the source database and the target database, and provides optimization suggestions to the change data capture module 1. This deeply coupled architecture design enables collaborative work between the modules, ensuring the efficient operation of the entire system.

[0063] The system provided in this embodiment of the invention supports horizontal scaling deployment. It enhances source-end data acquisition capabilities by increasing the number of instances of the change data capture module 1, and improves intermediate processing capabilities by increasing the number of instances of the stream processing module 3. Instances of each module communicate decoupled through message queues, supporting flexible capacity planning and elastic scaling.

[0064] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for real-time fusion and incremental synchronization of heterogeneous data sources, characterized in that, include: The data capture step involves connecting to multiple heterogeneous data sources via a source connector, parsing database transaction logs or message queues to obtain data change events. The heterogeneous data sources include relational databases and message middleware, and the data change events include operation type, primary key identifier, value before change, value after change, and transaction identifier. The event standardization step converts the data change event into a standardized message format, which includes a source identifier field, a table identifier field, an operation type field, a primary key field, a data payload field, a transaction identifier field, and a timestamp field, and generates a standardized change message, which is the data change event after being converted by the standardized message format. The stream processing step deduplicatizes and sorts the standardized change messages in chronological order, identifies transaction boundaries based on the transaction identifier field, aggregates standardized change messages belonging to the same transaction into transaction message groups, and outputs an ordered sequence of transaction message groups. The adaptive scheduling step collects the change rate of each data source and the write latency of the target end. When the number of pending transaction message groups exceeds the preset data backlog threshold, the number of consumer threads is increased. When the synchronization latency is lower than the preset latency recovery threshold, the number of consumer threads is reduced. The adjusted consumption parallelism is applied to the stream processing step. The consistency verification step involves periodically collecting source and target data digests, comparing the differences between the source and target data digests, generating a data consistency verification report, and triggering an incremental repair process when data inconsistency is detected.

2. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the change data capture step, the relational databases include MySQL, PostgreSQL, and Oracle, and the message middleware includes Kafka and RabbitMQ; parsing the database transaction logs includes parsing the binary logs of MySQL, the write-ahead logs of PostgreSQL, and the redo logs of Oracle.

3. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, The data backlog threshold ranges from 1,000 to 100,000 transaction message groups, and the delay recovery threshold ranges from 100 milliseconds to 5,000 milliseconds.

4. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, The verification period for the consistency verification step ranges from 1 minute to 60 minutes, and the data digest is calculated using the SHA-256 hash algorithm.

5. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the event standardization steps, different field mapping rules are used for different types of data sources. Specifically, the table name in a relational database is mapped to the table identifier field, and the topic name in a message middleware is mapped to the table identifier field. The data payload field uses a key-value pair structure to store the field values ​​before and after the change.

6. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the stream processing step, the deduplication process performs a hash calculation based on the combined value of the source identifier field, the table identifier field, the primary key field, and the transaction identifier field. When the hash value is repeated, the duplicate message that arrives later is discarded. The time-series sorting is based on the timestamp field and arranged in ascending order.

7. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the adaptive scheduling step, the adjustment range of the number of consumer threads is positively correlated with the ratio of the number of pending transaction message groups to the data backlog threshold; when the ratio is greater than the preset expansion trigger coefficient, the number of consumer threads increases to 1.25 to 1.5 times the current value; when the synchronization delay is lower than the delay recovery threshold for multiple consecutive sampling periods, the number of consumer threads decreases to 0.5 to 0.8 times the current value; wherein, the expansion trigger coefficient ranges from 1.5 to 3, and the number of sampling periods is 2 to 5.

8. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the consistency verification step, the incremental repair process includes: locating the primary key range of inconsistent data, rereading the complete data within the primary key range from the source end, overwriting the corresponding position on the target end, and updating the verification status to "repaired".

9. The method for real-time fusion and incremental synchronization of heterogeneous data sources according to claim 1, characterized in that, In the stream processing step, the method for identifying the transaction boundary includes: creating a new transaction message group when a standardized change message containing a transaction start marker is received, adding subsequent standardized change messages with the same transaction identifier to the transaction message group, and closing the transaction message group and marking it as committable when a standardized change message containing a transaction commit marker is received.

10. A system for real-time fusion and incremental synchronization of heterogeneous data sources, used to implement the method for real-time fusion and incremental synchronization of heterogeneous data sources as described in any one of claims 1-9, characterized in that, include: The change data capture module is used to connect to multiple heterogeneous data sources through the source connector, parse database transaction logs or message queues to obtain data change events, wherein the heterogeneous data sources include relational databases and message middleware, and the data change events include operation type, primary key identifier, value before change, value after change and transaction identifier; The event standardization module is used to convert the data change event into a standardized message format and generate a standardized change message. The stream processing module is used to deduplicatize and sort the standardized change messages in time, identify transaction boundaries based on transaction identifiers, and output an ordered sequence of transaction message groups. The adaptive scheduling module is used to collect the change rate of each data source and the write latency of the target end, and dynamically adjust the consumption parallelism according to the data backlog and synchronization latency. The consistency verification module is used to periodically collect data summaries from the source and target ends, compare the differences, and generate a data consistency verification report.

Images