Incremental data synchronization method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an incremental data synchronization method and device, equipment, a storage medium and a program product, and relates to the field of distribution. The method comprises the following steps: acquiring transaction logs recorded according to a system change number sequence from a main node; dividing the transaction log into a plurality of log segments according to the numerical value interval of the system change number, and executing a log mining operation on each log segment in parallel to extract change data; sorting the mined change data according to the system change number; and applying the sorted changed data to a local database of the standby node. According to the method, a segmented parallel processing mechanism based on the system change number is added, and the synchronization efficiency of massive transaction logs is improved.

Description

Technical Field

[0001] This application relates to the field of distributed systems, and more particularly to an incremental data synchronization method, apparatus, device, storage medium, and program product. Background Technology

[0002] As enterprise data volumes continue to grow and business demands for real-time data processing increase, traditional batch data synchronization methods are no longer sufficient to meet the demands of modern data processing with high throughput and low latency.

[0003] In current implementations, incremental data synchronization using log miners (such as Logminer) is typically employed to meet real-time synchronization requirements. This approach parses the transaction logs of the source database (such as the database's redo logs), extracts the data manipulation language (DML) operations, and applies these changes sequentially to the target database, thereby achieving incremental data synchronization.

[0004] However, this implementation method makes it difficult to ensure data consistency between the source and target ends. Summary of the Invention

[0005] This application provides an incremental data synchronization method, apparatus, device, storage medium, and program product to solve the technical problem of data synchronization consistency.

[0006] In a first aspect, embodiments of this application provide an incremental data synchronization method applied to a backup node in a distributed system, the distributed system further including a master node, the method comprising:

[0007] The transaction log is obtained from the master node and is recorded in the order of system change number.

[0008] The transaction log is divided into multiple log segments according to the numerical range of the system change number, and log mining operations are performed in parallel on each log segment to extract change data.

[0009] Sort the mined change data according to the system change number;

[0010] The sorted change data is applied to the local database of the backup node.

[0011] In this embodiment, by combining SCN-based log segmentation with parallel mining technology, efficient parallel processing is achieved during the data acquisition phase, overcoming the performance bottleneck of traditional serial parsing. Subsequently, an SCN-based global sorting mechanism ensures the correctness of data timing under parallel processing. Ultimately, a balance between high performance and high reliability is achieved, making it suitable for production environments with high requirements for data real-time performance and system throughput, such as distributed database systems for financial transactions and real-time analytics.

[0012] In one possible implementation, the transaction log is divided into multiple log segments based on the numerical range of the system change number, including:

[0013] Obtain the range of system change numbers corresponding to the transaction log;

[0014] Based on the preset interval size, the system change number range is divided into multiple consecutive numerical intervals;

[0015] Each numerical range corresponds to an independent log mining task.

[0016] This implementation achieves systematic optimization of log segmentation granularity through parameterized control of preset interval sizes. A fixed-length system change number interval division mechanism ensures the independence and parallelism of each log mining task. This approach balances parallel efficiency and system overhead. Specifically, the interval size must avoid increasing thread switching and management overhead due to being too small, while also preventing insufficient parallelism and underutilization of multi-core resources due to being too large. By configuring this parameter appropriately, the overall throughput of log collection can be improved while effectively controlling the additional resource consumption introduced by parallelization, achieving an effective balance between processing performance and resource utilization.

[0017] In one possible implementation, the method further includes:

[0018] Based on system load metrics and log generation rate, the preset interval size used to divide log segments is dynamically adjusted.

[0019] This implementation improves the adaptability of the log segmentation strategy by dynamically adjusting the interval size. Specifically, this approach monitors system load and log generation rate in real time, ensuring that the SCN segmentation granularity matches the current load, thus avoiding performance bottlenecks in dynamic scenarios caused by fixed segmentation strategies. For example, in sudden traffic surges, dynamic adjustment can quickly respond to log traffic spikes, thereby improving the system's adaptability and resource utilization.

[0020] In one possible implementation, the preset interval size is dynamically adjusted, including:

[0021] When an increase in the log generation rate of high-priority transactions is detected, the preset interval size is reduced;

[0022] When a decrease in the log generation rate of low-priority transactions is detected, the preset interval size is increased.

[0023] In this implementation, a priority-aware dynamic adjustment mechanism enables intelligent resource scheduling oriented towards business value. The system differentiates the segment granularity based on changes in the log generation rate of high- and low-priority transactions: narrowing the interval to accelerate processing when high-priority transactions surge, and widening the interval to reduce overhead when low-priority transactions slow down. This allows the system to prioritize low-latency synchronization of critical business processes under complex loads while globally optimizing resource utilization efficiency, thus achieving an adaptive balance between performance assurance and system efficiency.

[0024] In one possible implementation, before sorting the mined change data according to the system change number, the following is also included:

[0025] Mark the corresponding system change number for each transaction operation in the changed data;

[0026] Based on the marked system change number, perform transaction order verification on multiple transaction operations within the same transaction.

[0027] This implementation strengthens the timing constraints of log mining results by using system change number (SCN) marking and transaction order verification. By adding a transaction order verification step before sorting, this method avoids subsequent processing anomalies caused by out-of-order SCNs of transaction operations in the log mining results. For example, in cross-regional data synchronization in the telecommunications industry, transaction order verification can detect and correct out-of-order transaction operations caused by mining parallelism in advance, thereby improving the reliability of data synchronization.

[0028] In one possible implementation, each transaction operation in the changed data is marked with a corresponding system change number, including:

[0029] Identify high-priority transactions from the changed data and assign priority tags to them;

[0030] The sorting step determines the order of transaction operations based on the system change number and priority flag.

[0031] This implementation combines priority marking with sorting logic to enhance the synchronization priority of critical business data. By dynamically adjusting priority markings, it reduces the synchronization latency of high-priority transactions (such as financial transactions), thereby ensuring the continuity and reliability of core business operations in complex scenarios.

[0032] In one possible implementation, the mined change data is sorted according to the system change number, specifically including at least one of the following methods:

[0033] A priority queue is used to sort the system change numbers in ascending order;

[0034] The merge sort algorithm is used to merge and sort the mining results of multiple log segments.

[0035] In this implementation, the sorting efficiency of log mining results is optimized by selecting the appropriate sorting algorithm. Priority queues are suitable for scenarios with high real-time requirements, while merge sort is suitable for scenarios involving merging results from multiple threads. For example, in batch synchronization of a data warehouse, merge sort can efficiently merge partially ordered results from multiple threads, avoiding the high overhead of full sorting, thereby improving processing efficiency while ensuring data order.

[0036] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0037] The memory stores instructions that the computer executes;

[0038] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0039] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0040] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 This application provides a schematic diagram illustrating a scenario representing the data flow direction in its embodiments.

[0043] Figure 2 A flowchart illustrating an incremental data synchronization method provided in this application embodiment. Figure 1 ;

[0044] Figure 3 A schematic diagram illustrating a data mining and parsing scenario provided in an embodiment of this application;

[0045] Figure 4 A flowchart illustrating an incremental data synchronization method provided in this application embodiment. Figure 2 .

[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] It should be noted that the incremental data synchronization method, apparatus, device, storage medium, and program product provided in this application can be used in the distributed field, or in any field other than distributed fields. This application does not limit the application field of the incremental data synchronization method, apparatus, device, storage medium, and program product.

[0049] Specific application scenarios for this application may include enterprise-level real-time database replication, distributed system data consistency assurance, data warehouse construction, and disaster recovery system setup. With the surge in enterprise data volume and the increasing demands for real-time data processing, traditional batch data synchronization methods can no longer meet the needs of high-throughput, low-latency data processing.

[0050] For example, in the financial industry, real-time data from trading systems needs to be immediately synchronized to risk control and auditing systems to ensure the timeliness and compliance of business decisions; in e-commerce, user behavior data needs to be synchronized to recommendation systems in real time to achieve accurate personalized recommendations; and in IoT applications, data generated by massive numbers of terminal devices needs to be aggregated in real time to a central database for analysis. All of these scenarios place high demands on data synchronization technology in terms of throughput, low latency, and strong consistency.

[0051] To meet the aforementioned real-time requirements, incremental data synchronization technology based on database log analysis is typically employed. The core of this technology lies in parsing database logs (such as redo logs and archive logs) to capture data changes in the source database, thereby achieving low-latency real-time data replication. Taking Logminer, a native database log analysis tool, as an example, the synchronization process of this technology can generally be divided into three stages:

[0052] Initialization phase: Logminer analyzes the source database logs to determine the starting system change number (SCN) for incremental synchronization as the synchronization reference point, and completes the full loading of existing data on the source and target sides to achieve initial data alignment.

[0053] Incremental synchronization phase: Based on the starting system change number (SCN), the Logminer tool is used to parse the online or archived logs of the source database, capture the DML operations (including insert, update and delete operations) recorded in the logs, convert them into structured data that the synchronization system can recognize, synchronize them to the target end and execute them to maintain data consistency between the source and target ends.

[0054] Verification phase: To ensure the integrity of the synchronization process, this phase is executed in parallel with the incremental synchronization phase. The synchronization data between the source and target ends is compared and verified periodically to confirm that there is no data loss or error.

[0055] The specific implementation steps of the incremental synchronization phase are as follows:

[0056] Log parsing: The Logminer tool extracts SQL statements from the source database logs. The parsing module of the synchronization system obtains the metadata (including column names, data types, field lengths, etc.) of the target database table through the JDBC interface, parses the SQL statements based on the metadata, and converts them into the structured data format preset by the synchronization system.

[0057] Data transmission and application: The transmission module of the synchronization system sends structured data to the target end through a private communication protocol. After receiving the structured data, the synchronization component at the target end restores it to the SQL statements supported by the target database and executes them to complete the synchronization and application of incremental data.

[0058] In terms of specific implementation architecture, a limited parallel model with multi-threaded cooperation is typically adopted. For example, data synchronization tools create dedicated threads to perform tasks. The following is a specific example illustrating this implementation architecture:

[0059] Reader thread: Responsible for calling the Logminer tool's interface, reading log segment data from the source database, and generating original records of data changes.

[0060] Parser thread: Responsible for obtaining metadata of the target database table through JDBC connection, parsing the SQL statements in the raw records generated by the read thread, and converting them into the structured data mentioned above.

[0061] Extractor thread: Responsible for encapsulating the structured data processed by the parsing thread into a preset data structure or message queue, and providing it to the transmission module for subsequent data transmission.

[0062] To better understand the above implementation process, the following will combine... Figure 1 The process of incremental data synchronization will be introduced. Figure 1This is a schematic diagram illustrating a data flow scenario provided in an embodiment of this application. For example... Figure 1 The diagram illustrates the incremental data transmission and filtering logic from the source to the target. The specific steps are as follows:

[0063] Data source: After the source database performs DML operations, it generates corresponding incremental data (i.e., data change records) and writes them to the database log.

[0064] Source synchronization component: It collects incremental data from the source database logs through a reading thread and passes the collected incremental data to the filtering stage.

[0065] Filtering step: The collected incremental data is filtered to remove invalid data (such as data with incorrect format) and redundant data (such as data transmitted repeatedly), so as to obtain the filtered valid incremental data.

[0066] Target-side synchronization component: Receives valid incremental data, completes data parsing and encapsulation through the collaboration of parsing thread and extraction thread, and applies it to the target-side database.

[0067] Final result: The target database obtains incremental data consistent with the source database, realizing data synchronization between primary and backup nodes or across databases.

[0068] The above architecture is the basic architecture for database data synchronization scenarios (such as master-slave synchronization, off-site backup, etc.). Through the links of collection, filtering, and transmission, it achieves the consistency of data between the source and target ends, and improves the effectiveness and accuracy of synchronized data through the filtering process.

[0069] However, the solution described above has the following technical problems:

[0070] First, the core of the above solution relies on native single-threaded log parsing tools such as Logminer. Under high database load or when processing complex transactions, the log generation speed easily exceeds the single-threaded parsing capability, causing synchronous pipeline blockage and accumulated data latency. The overall system throughput is limited by this serial processing mode, making it difficult to meet the needs of high real-time business scenarios.

[0071] Secondly, although existing architectures attempt to improve performance by introducing multi-threading (e.g., read, parse, and extract threads) for pipelined collaboration, they fail to guarantee the parsing and transmission order of data change records within the same transaction. When multiple threads process different records of the same transaction in parallel, these records may be executed out of order on the target end, thus violating the atomicity of the transaction. This can lead to inconsistencies between the data state on the target and source ends, manifesting as consistency issues such as primary key conflicts and incorrect associations.

[0072] Furthermore, the existing fixed-pipeline thread model is a static resource allocation mechanism that cannot adaptively adjust to the dynamic fluctuations in log load. This may lead to an imbalance in system resource utilization, where resources are idle under low load, while under high load, new performance bottlenecks arise due to uneven thread task allocation and intensified lock contention. This lack of dynamic adjustment makes it difficult for the system to effectively cope with the ever-increasing data processing pressure, further limiting the improvement of synchronization performance.

[0073] The incremental data synchronization method provided in this application aims to solve the aforementioned technical problems of existing technologies. Based on the parallel processing mechanism of Logminer, it achieves collaborative optimization of multi-threaded log mining and parsing, combined with a conditional locking mechanism to ensure transaction order consistency, thereby overcoming the performance bottleneck of Logminer and improving the real-time performance and reliability of incremental data synchronization. The core of this method lies in the following aspects:

[0074] On the one hand, the single-threaded Logminer log mining is split into multi-threaded parallel execution, and the log mining interval is divided by SCN (System Change Number) to improve log parsing efficiency. On the other hand, multi-threaded parallel processing of SQL statements is introduced in the parsing phase, and the parsing order of transaction data is dynamically controlled by the conditional locking mechanism, so that the changed data in the same transaction maintains atomicity and consistency on the target end.

[0075] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0076] First, combine Figure 2 The incremental data synchronization method provided in the embodiments of this application will be described. Figure 2 A flowchart illustrating an incremental data synchronization method provided in this application embodiment. Figure 1 This method is applied to distributed database systems, which include one master node and at least one slave node. The master node is responsible for processing business logic and generating transaction logs, while the slave node is used to synchronize data changes from the master node in real time. This method is executed on the slave node side to achieve efficient and orderly incremental data synchronization. Figure 2 As shown, the method includes the following steps:

[0077] S201. Obtain transaction logs from the master node.

[0078] The standby node continuously retrieves the generated database transaction logs from the master node through the database's log transmission mechanism. The transaction logs (also known as redo logs or archive logs) are recorded strictly in order of system change number (SCN). The SCN is a monotonically increasing logical timestamp within the database, uniquely identifying the change order and version of each committed transaction.

[0079] For example, if the commit SCN of transaction A is 1001 and the commit SCN of transaction B is 1002, then in the log, the change record of transaction B is usually located after the record of transaction A.

[0080] S202. Divide the transaction log into multiple log segments according to the numerical range of the system change number, and perform log mining operations in parallel on each log segment to extract change data.

[0081] This step's processing flow may include two key steps: log segmentation and parallel mining. The specific implementation method is as follows:

[0082] Log Segmentation: The standby node analyzes the acquired transaction logs to determine the SCN range it covers (e.g., SCN range is 1000 to 2000). Based on a preset or dynamically calculated interval size, this SCN range is divided into multiple consecutive, non-overlapping numerical intervals. For example, if the preset interval size is 200, then the SCN range 1000-2000 is divided into five log segments: [1000, 1200), [1200, 1400), [1400, 1600), [1600, 1800), and [1800, 2000]. Each log segment corresponds to a continuous fragment of the original log data.

[0083] Parallel logging: An independent log mining task, such as a separate thread or process, is launched for each partitioned log segment. Each mining task uses a database log parsing tool to parse its assigned log segment in parallel. The core of log mining is extracting readable, structured database change operations from binary or structured log records. These operations include SQL statements such as INSERT, UPDATE, and DELETE, as well as information such as the SCN, timestamp, and involved tables and rows for each operation. This extracted structured information constitutes the change data.

[0084] Since the log segments are contiguous and mutually exclusive on the SCN, multiple mining tasks can be executed in parallel without conflict, fully utilizing the computing power of multi-core processors and improving the throughput of the log parsing (i.e., data collection) stage.

[0085] S203. Sort the extracted change data according to the system change number.

[0086] After the parallel mining step is completed, multiple sets of change data from various log mining tasks are obtained. Since this data is generated in parallel, it may not be arranged in global SCN order. Therefore, it is necessary to globally sort all the mined change data according to the SCN associated with each change data to reconstruct the timeline that is completely consistent with the transaction commit order of the master node.

[0087] In one possible implementation, sorting can be achieved using a priority queue. For example, by inserting changed data into a min-heap using SCN as the priority key, and then popping data in sequence, an ordered sequence can be obtained by sorting data in ascending order of SCN.

[0088] In another possible implementation, considering that the data within each log segment is naturally ordered due to the continuity of the SCN (i.e., the output of each log mining task is already sorted in ascending order of SCN), sorting can be performed using, for example, a merge sort algorithm. Specifically, the ordered change data sequence mined from each log segment is regarded as an ordered subsequence, and then these ordered subsequences are merged into a globally ordered change data stream through multi-way merge sort.

[0089] The purpose of sorting is to ensure that subsequent steps apply these data changes in the correct logical order, consistent with that of the master node, which is key to guaranteeing eventual consistency of data on the backup nodes.

[0090] S204. Apply the sorted change data to the local database of the backup node.

[0091] The globally sorted and time-sequentially correct changed data is committed or applied sequentially to the local database of the standby node. This process is typically accomplished through the database's SQL execution engine, which executes the INSERT, UPDATE, DELETE, and other SQL statements contained in the sorted changed data in sequence. Because the change order is completely consistent with the primary node, the data state of the standby node will eventually be synchronized with the primary node.

[0092] To facilitate understanding of the above process, a specific example will be used below to illustrate the content described above.

[0093] For example, suppose the master node generates transaction logs with SCNs from 1000 to 1400.

[0094] In S202, the transaction log is divided into log segments [1000, 1200) and [1200, 1400], and two threads are started to mine in parallel. Thread 1 mines changes with SCN 1005 (INSERT) and 1100 (UPDATE); thread 2 mines changes with SCN 1205 (DELETE) and 1300 (INSERT).

[0095] In S203, the four changes are sorted according to the SCN, resulting in the following order: 1005 (INSERT) -> 1100 (UPDATE) -> 1205 (DELETE) -> 1300 (INSERT).

[0096] Finally, in S204, these SQL operations are executed on the standby node database in strict order to ensure that the standby node data is completely consistent with the state of the primary node at SCN 1400.

[0097] The incremental data synchronization method provided in this embodiment combines SCN-based log segmentation with parallel mining technology to achieve efficient parallel processing during the data acquisition phase, overcoming the performance bottleneck of traditional serial parsing. Subsequently, an SCN-based global sorting mechanism ensures the correctness of data timing under parallel processing. Ultimately, it achieves a balance between high performance and high reliability, making it suitable for production environments with high requirements for data real-time performance and system throughput, such as distributed database systems for financial transactions and real-time analytics.

[0098] The following will combine Figure 3 The incremental data synchronization method provided in the embodiments of this application will be further described. Figure 3 This is a schematic diagram illustrating a scenario of data mining and parsing provided in an embodiment of this application.

[0099] like Figure 3 As shown, this is achieved through a collaborative architecture of "parallel mining - queue buffering - ordered processing". The core of this architecture lies in using multi-threaded parallelism to improve throughput and using an intermediate queue to decouple and buffer the production and consumption stages. Its specific collaborative logic includes the following three stages:

[0100] Parallel mining phase: This corresponds to the parallel execution of log mining operations in step S202. For example... Figure 3 As shown on the left side of the architecture, the system maintains a mining thread group containing multiple independent mining threads. For example, the diagram shows three mining threads: mining thread 1 to mining thread 3. Each mining thread is assigned an independent log segment (SCN interval). These threads work concurrently, each parsing its assigned log segment and performing log mining operations. After each thread completes mining, it submits the resulting mining results (i.e., changed data) to a shared mining result queue via a push operation. This multi-threaded parallel design fully utilizes the computing resources of the server's multi-core CPUs, transforming the originally serial log parsing task into a parallel task, thereby improving the overall throughput of the data acquisition phase.

[0101] Queue caching: The mining results queue is a key buffer component in this architecture, connecting the parallel mining process with subsequent processing. This queue receives changed data from all mining threads, serving two purposes: firstly, it prevents a mining thread from running out of storage space (i.e., blocking) due to its fast processing speed; secondly, it provides a stable and continuous data source for downstream parsing threads, ensuring a smooth running rhythm even if mining speed fluctuates, thus achieving decoupling and matching between upstream and downstream processing capabilities.

[0102] Parallel parsing stage: This corresponds to the sorting logic and subsequent processing preparation in step S203. For example... Figure 3 As shown on the right side of the architecture, the system maintains a parsing thread group, which also contains multiple parsing threads, such as the three parsing threads shown in the diagram. These threads retrieve changed data from the mining results queue through retrieval operations. To ensure the global transaction order, the system needs to sort the changed data at this point based on the System Change Number (SCN).

[0103] In one possible implementation, data can be sorted before entering the queue or after being retrieved by the parsing thread; in another possible implementation, the queue itself can be designed as a priority queue, with its dequeue order arranged in ascending order of SCN, thereby achieving ordered consumption.

[0104] The parsing thread processes the acquired (sorted or unsorted) changed data in parallel, performing tasks such as transaction integrity checks and data format conversions. Once processing is complete, the results are passed to subsequent modules. Figure 3 The module shown in the middle is the data application layer module, which is ultimately used for database submission in step S204.

[0105] Optionally, in step S202, in order to divide the transaction log into multiple log segments that can be processed in parallel, the system adopts a division strategy based on the system change number (SCN) value range.

[0106] In practice, the system first determines the complete SCN range covered by the transaction log to be processed, for example by parsing the log header or scanning key records to obtain the starting SCN and the ending SCN, thereby clarifying a processing range such as [1000, 2500].

[0107] Subsequently, based on a configurable preset parameter, namely the interval size, this continuous range is evenly divided into multiple non-overlapping numerical sub-intervals. For example, if the preset interval size is 500, the above range will be divided into three continuous intervals: [1000, 1500), [1500, 2000), and [2000, 2500]. Each divided numerical interval corresponds to an independent log mining task, and the system will allocate an independent execution thread to each task, enabling it to mine log data within its respective interval in parallel. This division method not only ensures clear and conflict-free data boundaries between tasks but also provides a structured input foundation for subsequent parallel mining and merge sorting.

[0108] By parameterizing the preset interval size, a systematic optimization of the log segmentation granularity is achieved. A fixed-length system change number interval division mechanism ensures the independence and parallelism of each log mining task. This implementation balances parallel efficiency and system overhead. Specifically, the interval size must avoid increasing thread switching and management overhead due to being too small, while also preventing insufficient parallelism and underutilization of multi-core resources due to being too large. Through proper configuration of this parameter, the overall throughput of log collection can be improved while effectively controlling the additional resource consumption introduced by parallelization, achieving an effective balance between processing performance and resource utilization.

[0109] Optionally, before dividing the system change number range into multiple consecutive numerical intervals according to the preset interval size, a dynamic adjustment operation is first performed to determine the interval size to be used for this division.

[0110] In one possible implementation, the aforementioned adjustments could be performed based on real-time collected system load metrics and log generation rate. Specifically, system load metrics reflect the current resource strain on the standby node, such as CPU utilization, memory usage, or I / O load; while log generation rate reflects the data input pressure from the master node. By comprehensively analyzing the real-time status and historical trends of these two types of metrics, the preset interval size parameter is dynamically calculated and updated.

[0111] This preliminary adjustment step ensures that the interval size parameter used each time log partitioning is no longer a fixed value or a simple configuration, but an optimized result derived from real-time evaluation. The aim is to allow the partitioning strategy to adapt to the current system operating environment and processing pressure, providing a more reasonable task granularity in terms of parallelism and single-task processing capacity for the subsequent parallel mining phase, thereby achieving adaptive optimization of the overall processing flow from the source.

[0112] By dynamically adjusting the interval size, the adaptability of the log segmentation strategy is improved. Specifically, this approach monitors system load and log generation rate in real time, ensuring that the SCN segmentation granularity matches the current load, thus avoiding performance bottlenecks in dynamic scenarios where fixed segmentation strategies fail. For example, in sudden traffic surges, dynamic adjustment can quickly respond to log traffic spikes, thereby improving the system's adaptability and resource utilization.

[0113] Optionally, during the dynamic adjustment of the preset interval size, a transaction priority awareness mechanism can be introduced to ensure resource allocation for critical business data. Specifically, the system can distinguish the priorities of different transactions, for example, marking financial transaction transactions as "high priority" and log analysis transactions as "low priority," and monitoring the log generation rate of high-priority and low-priority transactions respectively.

[0114] Based on this, the dynamic adjustment strategy performs the following operations:

[0115] When an increase in the log generation rate of high-priority transactions is detected, it indicates a rise in the volume of critical business data. To enable these high-priority transactions to be processed quickly and reduce synchronization latency, the system proactively reduces the preset interval size used to divide log segments. In other words, the same SCN range is divided into more and finer log segments, thereby creating more parallel data mining tasks within the system. This allows the system to allocate more computing threads and resources to process log data containing high-priority transactions, achieving accelerated processing and prioritized resource allocation for critical business operations.

[0116] When a decrease in the log generation rate of low-priority transactions is detected, it indicates that the data pressure on non-critical business operations has eased. At this point, the system will correspondingly increase the preset interval size. This results in the SCN range being divided into fewer, coarser log segments, thereby reducing the total number of concurrent mining tasks in the system. This reduces the system overhead caused by excessive parallel tasks, such as thread scheduling, context switching, and synchronization management, allowing the freed-up computing resources to more effectively serve high-priority tasks or reduce overall energy consumption. This achieves improved overall system resource utilization efficiency while ensuring core business operations are maintained.

[0117] Through a priority-aware dynamic adjustment mechanism, intelligent resource scheduling oriented towards business value is achieved. The system differentiates the segment granularity based on changes in the log generation rate of high- and low-priority transactions: narrowing the interval to accelerate processing when high-priority transactions surge, and widening the interval to reduce overhead when low-priority transactions slow down. This enables the system to prioritize low-latency synchronization of critical business processes under complex loads while globally optimizing resource utilization efficiency, thus achieving an adaptive balance between performance assurance and system efficiency.

[0118] Optionally, before submitting the changed data obtained from parallel mining to the global sorting step, the system will first perform a check of the logical order within each transaction to ensure that the operation sequence of each independent transaction is logically correct. This check specifically includes the following operations:

[0119] First, each transaction operation in the changed data is labeled with its corresponding system change number. Since the changed data output by parallel mining is usually based on transaction operations (such as INSERT, UPDATE, DELETE, COMMIT), the system accurately records the system change number for each such operation unit. This labeling process assigns an accurate logical timestamp to each operation, establishing a correlation between the operation and the global time sequence.

[0120] Next, based on the marked system change number, the system performs transaction order verification on multiple operations within the same transaction. The system aggregates all operations belonging to the same database transaction (usually identified by the same transaction ID) and then checks whether their marked SCN values ​​exhibit a logically correct increasing relationship. For example, a complete transaction might contain an UPDATE operation and a COMMIT operation. A correct logical order requires that the SCN of the UPDATE operation should be less than the SCN of its COMMIT operation. The system verifies such constraints, and if it finds an anomaly in the SCN order of operations within a transaction (e.g., the SCN of the COMMIT operation is actually smaller), it can promptly detect data anomalies or parsing errors.

[0121] By using system change number (SCN) marking and transaction order verification, the timing constraints of log mining results are strengthened. This method, by adding a transaction order verification step before sorting, avoids subsequent processing anomalies caused by out-of-order SCNs of transaction operations in the log mining results. For example, in cross-regional data synchronization in the telecommunications industry, transaction order verification can detect and correct out-of-order transaction operations caused by mining parallelism in advance, thereby improving the reliability of data synchronization.

[0122] Optionally, in the data synchronization system, high-priority transaction identification rules are pre-configured, core transactions involving fund transfers and order payments are preset as high-priority transactions, while non-core transactions such as user registration and log recording are preset as low-priority transactions, and priority identifier metadata corresponding to the transaction priority level is defined.

[0123] The system collects change data generated by the source database, automatically marks each transaction operation with the corresponding System Change Number (SCN), and then identifies high-priority transactions by parsing transaction SQL statement keywords, as well as related business tables, and in combination with preset rules, assigning high-priority identifiers to them and low-priority transactions with low-priority identifiers, forming structured data containing priority identifiers.

[0124] The sorting phase adopts a priority-first, SCN-complete logic: first, high-priority transactions are placed before low-priority transactions, and then transactions of the same priority are sorted according to their SCN values ​​to ensure that the execution order of transactions within the same priority is consistent with the source.

[0125] For example, a high-priority transaction C (SCN 10005, with a high-priority identifier) ​​has a higher sorting priority than a low-priority transaction D (SCN 10006, with a low-priority identifier). Even if transaction D has a larger SCN value, transaction C is still synchronized first, thereby ensuring the synchronization priority of core financial business data and taking into account both business continuity and data consistency.

[0126] By combining priority marking with sorting logic, the synchronization priority of critical business data is improved. This approach reduces the synchronization latency of high-priority transactions (such as financial transactions) through dynamic adjustment of priority marking, thereby ensuring the continuity and reliability of core business in complex business scenarios.

[0127] Optionally, the implementation of sorting the changed data according to the system change number can include, for example, the following implementation methods:

[0128] Method 1: Use a priority queue to sort by SCN in ascending order.

[0129] After the system starts multi-threaded parallel mining of source database logs, each thread stores the mined change data into a priority queue in real time. The queue is based on the SCN value for priority determination: whenever new change data is added to the queue, the queue automatically adjusts the element order according to the ascending order of SCN; when retrieving data later, the change data with the smallest SCN in the queue is extracted each time.

[0130] For example, if thread 1 finds a record with SCN=10003 and thread 2 finds a record with SCN=10001, after both are enqueued, the queue will place the record with SCN=10001 at the head of the queue to ensure that the output data is arranged in ascending order of SCN.

[0131] Method 2: Use the merge sort algorithm to merge the results of multiple log segments.

[0132] When multiple threads mine logs according to different SCN intervals, each thread first sorts the results of the log segment it is responsible for in ascending order of SCN. For example, the result of thread A is [SCN=10001,10003], and the result of thread B is [SCN=10002,10004]). Then, the merge sort algorithm is used to merge these locally ordered subsequences into a globally ordered sequence: the SCN value of the current first element of each subsequence is compared in turn, and the element with the smallest value is added to the global sequence, finally obtaining the complete ordered change dataset [SCN=10001,10002,10003,10004].

[0133] In practical applications, the sorting method can be selected based on the number of threads and the amount of data for log mining: when the number of threads is small and the real-time requirements of the data are high, a priority queue is used for dynamic sorting; when the number of threads is large and the amount of data is large, merge sort is used to merge local results, thereby adapting to the sorting requirements of different scenarios and providing a globally ordered data source for subsequent transaction parsing.

[0134] The sorting efficiency of log mining results was optimized by selecting the appropriate sorting algorithm. Priority queues are suitable for scenarios with high real-time requirements, while merge sort is suitable for scenarios involving merging results from multiple threads. For example, in batch synchronization of a data warehouse, merge sort can efficiently merge partially ordered results from multiple threads, avoiding the high overhead of full sorting, thereby improving processing efficiency while ensuring data order.

[0135] To facilitate understanding of the overall process of the above embodiments, the following will combine... Figure 4 The overall implementation process of the embodiments of this application is further described below. Figure 4 A flowchart illustrating an incremental data synchronization method provided in this application embodiment. Figure 2 Its core is to achieve high-throughput, ordered incremental data synchronization through a full-chain process of "log segmentation and mining → data sorting → transaction parsing → synchronization and application". The specific steps are as follows:

[0136] Log segment mining: The process starts with "starting the segment mining task". First, it performs operations such as "getting the current maximum value of SCN" and "calculating the segment SCN interval" to split the source database log into multiple independent intervals according to SCN. Then, it assigns a corresponding log mining thread to each interval, and each thread performs operations such as "calling the Logminer interface to read logs" and "generating original records of change data" in parallel to complete the parallel collection of log data.

[0137] Data sorting: The results of each mining thread will enter the unparsed queue. Then the system sorts the changed data based on SCN: the order can be dynamically adjusted by priority queue according to SCN ascending order, or merge sort can be used to merge the locally ordered results of multiple threads to ensure that the output data is a globally SCN ordered changed dataset; the sorted results will be stored in the "parsed result cache" as an ordered data source for subsequent transaction parsing.

[0138] Transaction parsing: After retrieving ordered modified data from the cache, the parsing thread first determines whether it is a new transaction and obtains the current transaction's SCN. Then, it controls the transaction order by checking whether the preceding SCN has been completed. If the transaction corresponding to the preceding SCN has not been completed, the current thread waits; if it has been completed, it parses the SQL statement, encapsulates it into structured data, and stores the result in the reorder cache. At the same time, it records the running transaction SCN to mark the processing progress.

[0139] Synchronization Application: When a transaction commit or rollback event is detected, the system processes the data in the reordering cache: if it is a commit, the transaction data is written to the target database to complete synchronization; if it is a rollback, the corresponding transaction data is deleted from the cache and the waiting thread is woken up. Finally, the target database obtains incremental data consistent with the source database, completing the data synchronization between the primary and backup nodes.

[0140] Overall, Figure 4 The process improves efficiency through parallel mining, ensures order through SCN sorting, and ensures consistency through transaction management. This collaborative design not only solves the performance bottleneck of traditional single-threaded synchronization, but also meets the real-time and consistency requirements of data synchronization in high-concurrency scenarios.

[0141] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0142] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0144] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0145] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0149] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0150] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0151] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0152] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0153] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0154] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0155] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An incremental data synchronization method, characterized in that, The method, applied to a backup node in a distributed system, which also includes a master node, comprises: The transaction log is obtained from the master node, and the transaction log is recorded in the order of system change number; The transaction log is divided into multiple log segments according to the numerical range of the system change number, and log mining operations are performed in parallel on each log segment to extract change data. The mined change data is sorted according to the system change number; The sorted changed data is then applied to the local database of the backup node.

2. The method according to claim 1, characterized in that, The step of dividing the transaction log into multiple log segments according to the numerical range of the system change number includes: Obtain the range of system change numbers corresponding to the transaction log; Based on a preset range size, the system change number range is divided into multiple consecutive numerical ranges; Each of the numerical ranges corresponds to an independent log mining task.

3. The method according to claim 1, characterized in that, The method further includes: Based on system load metrics and log generation rate, the preset interval size used to divide log segments is dynamically adjusted.

4. The method according to claim 3, characterized in that, The dynamic adjustment of the preset interval size includes: When an increase in the log generation rate of high-priority transactions is detected, the preset interval size is reduced; When a decrease in the log generation rate of low-priority transactions is detected, the preset interval size is increased.

5. The method according to any one of claims 1 to 4, characterized in that, Before sorting the mined change data according to the system change number, the method further includes: Each transaction operation in the changed data is marked with its corresponding system change number; Based on the marked system change number, perform transaction order verification on multiple transaction operations within the same transaction.

6. The method according to claim 5, characterized in that, The system change number corresponding to each transaction operation marker in the changed data includes: High-priority transactions are identified from the changed data, and priority tags are assigned to these high-priority transactions. The sorting step determines the sorting order of transaction operations based on the system change number and the priority flag.

7. The method according to any one of claims 1 to 4, characterized in that, The step of sorting the mined change data according to the system change number specifically includes at least one of the following methods: The system change number is sorted in ascending order using a priority queue. The merge sort algorithm is used to merge and sort the mining results of multiple log segments.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images