A method and apparatus for data synchronization and failure recovery

Abstract

The present application relates to the technical field of data synchronization, and provides a data synchronization and fault recovery method and device.The present application combines a reading component, a conversion component and a loading component to design and obtain a required data synchronization task in a componentized manner, thereby improving the scalability of the data synchronization system;the reading component obtains transaction logs involved in the data synchronization task, and transmits the transaction logs to the conversion component according to the result of concurrent conflict detection;the conversion component converts the transaction logs to obtain target logs and transmits the target logs to the loading component;finally, the loading component loads the target logs into a database;when a loading fault occurs, the target logs are recovered according to the result of concurrent conflict detection.The present application supports rich data conversion operations, realizes concurrent conflict detection of transactions, optimizes the loading fault recovery strategy, and solves the problems of low usability of data conversion operations, poor data synchronization performance, and data loss when a loading fault occurs.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a method and apparatus for data synchronization and fault recovery. Background Technology

[0002] Database management systems (DBMS) use log files to store records of data changes. When events or actions are performed, including data modification operations such as Data Manipulation Language (DML) operations (writing, modifying, and deleting data) and Data Definition Language (DDL) operations (creating and modifying tables), the DBMS appends these changes to the log file. In the event of an unexpected DBMS crash, data can be recovered from the log file upon restart, ensuring data consistency. This was the original purpose of introducing log files into DBMSs. The log file contains all the details of data changes, which can be used for data synchronization, especially incremental data synchronization. Synchronizing data based on the log file fundamentally solves the challenges of incremental data synchronization.

[0003] Existing incremental data synchronization systems based on transaction logs are generally limited to a source-destination architecture, comprising two service components: a source and a destination. The source acquires transaction event logs and sends them to the destination. The destination converts the transaction event logs into Structured Query Language (SQL) or other commands, submitting them to the destination database's server for execution to achieve data consistency with the source database and complete data synchronization. This architecture is typically used for data synchronization tasks with one master and one backup or one master and multiple backups where data is completely consistent. It does not support data transformation operations or relies on complex attribute file configurations to achieve certain forms of data transformation, making it inconvenient to operate and maintain. Existing technologies design data synchronization tasks through configuration files or abstract data transformation function interfaces. For complex data synchronization tasks, there is a lack of independent and flexible design support for task design.

[0004] Because event logs belonging to different transactions are interleaved in the log files, existing technologies perform data synchronization according to the serial commit order in the event logs to ensure that no errors occur when transactions operate on the same data object, i.e., to ensure data consistency and correctness between the source and destination databases. However, serial execution is detrimental to data synchronization performance, affects the real-time nature of data synchronization, greatly reduces synchronization efficiency, and makes the data synchronization system less practical.

[0005] Furthermore, when a loading failure occurs, the entire data synchronization task is often stopped directly. Since the source server's log files are generally only kept for a limited time, the data will be truncated when the timeout period expires or the set log file size threshold is exceeded. If the log files are not parsed and the event information within them is not read in a timely manner, log file truncation will lead to data loss.

[0006] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a method and apparatus for data synchronization and fault recovery. Its purpose is to design complex data synchronization tasks in a modular way, so that the data synchronization system has strong scalability, realizes the detection of concurrent conflicts of transactions and optimizes the loading fault recovery strategy. It solves the problems of existing data synchronization systems based on transaction log increments, which have low usability for data conversion operations through attribute file configuration, poor performance of data synchronization using serial execution methods, and data loss caused by stopping the entire data synchronization task when a loading fault occurs.

[0008] The present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for data synchronization and fault recovery, comprising:

[0010] The required data synchronization task can be obtained by combining at least one read component and at least one load component; or, the required data synchronization task can be obtained by combining at least one read component, at least one transformation component, and at least one load component.

[0011] The data synchronization task's transaction logs are obtained by the reading component. Depending on whether the transaction logs have been stored in the database, conflicting transactions are selectively released or concurrent conflict detection is performed on the transaction logs. Based on the result of the conflicting transaction release or the result of the concurrent conflict detection, the transaction logs are transmitted to all subsequent components. The subsequent components of the reading component are the transformation component and the loading component.

[0012] When the subsequent component of the reading component is a transformation component, the transaction log is transformed by at least one transformation component to obtain the target log, and the target log is transmitted to the loading component; the target log is then stored in the database by the loading component.

[0013] When the subsequent component of the reading component is the loading component, the target log is loaded into the database through the loading component; when a loading failure occurs, the target log is restored according to the result of the concurrent conflict detection.

[0014] Further, the step of obtaining the transaction logs involved in the data synchronization task through the reading component, selectively performing conflict transaction release or concurrent conflict detection on the transaction logs based on whether the transaction logs have been stored in the database, and transmitting the transaction logs to all subsequent components based on the result of the conflict transaction release or the result of the concurrent conflict detection includes:

[0015] After obtaining the complete transaction log of the transaction to be detected, the transaction to be detected is taken as the last transaction in the complete transaction sequential linked list, and the transaction to be detected is taken as the last ready transaction in the transmission ready queue.

[0016] The completed transaction is obtained from the loading completion notification queue, removed from the complete transaction sequence list, and designated as the first ready transaction in the transmission ready queue; wherein, the complete transaction sequence list includes transactions being transmitted and transactions in the conflicting transaction queue;

[0017] Ready transactions are retrieved sequentially from the transmission ready queue, and it is determined whether the transaction log of the ready transaction has been entered into the database.

[0018] When the retrieved ready transaction is a completed transaction, the transaction that has a concurrent conflict with the ready transaction is identified from the conflict transaction queue, and the conflict transaction is released for the completed transaction; the transaction log of the transaction that has a concurrent conflict with the ready transaction is transmitted to all subsequent components.

[0019] When a ready transaction is not a completed transaction, it is determined whether the ready transaction has a concurrent conflict with the transactions in the complete transaction sequence list, so as to perform concurrent conflict detection on the ready transaction; based on the result of the concurrent conflict detection, the ready transaction is selectively attached to the conflict transaction queue, or the transaction log of the ready transaction is transmitted to all subsequent components.

[0020] Furthermore, when the retrieved ready transaction is not a completed transaction, it is determined whether the ready transaction has a concurrent conflict with transactions in the complete transaction sequence list, so as to perform concurrent conflict detection on the ready transaction; based on the result of the concurrent conflict detection, selectively attaching the ready transaction to the conflict transaction queue, or transmitting the transaction log of the ready transaction to all subsequent components includes:

[0021] Starting from the last transaction in the complete transaction sequence list, compare the transactions in the complete transaction sequence list with the current ready transaction in reverse order to determine whether there is any intersection between the data object operated by the current ready transaction and the data object operated by the transactions in the complete transaction sequence list.

[0022] When there is an intersection, stop judging the current ready transaction with the transactions that have not been compared in the complete transaction sequence list, record the transaction ID of the transaction that has an intersection with the current ready transaction as the ID to be released, make the current ready transaction carry the ID to be released, and attach the current ready transaction to the conflict transaction queue.

[0023] When there is no intersection, the current ready transaction is stored in the execution transaction queue, so that the transaction log of the current ready transaction is transmitted to all subsequent components according to the execution transaction queue.

[0024] Further, when the retrieved ready transaction is a completed transaction, the transaction that has a concurrent conflict with the ready transaction is determined from the conflicting transaction queue, and the completed transaction is released due to the conflict; transmitting the transaction log of the transaction that has a concurrent conflict with the ready transaction to all subsequent components includes:

[0025] Starting from the first transaction in the conflict transaction queue, the pending release ID carried by each transaction in the conflict transaction queue is compared with the transaction ID of the completed transaction in order to determine whether the completed transaction causes a conflict or blockage to the transactions in the conflict transaction queue.

[0026] When the ID to be released is the same as the transaction ID of the completed transaction, the transaction log corresponding to the ID to be released is transmitted to all subsequent components, and the ID to be released is stored in the execution transaction queue.

[0027] After all transactions in the conflict transaction queue have been evaluated, the completed transaction is removed from the execution transaction queue.

[0028] Furthermore, the step of restoring the target log based on the result of the concurrent conflict detection when a loading failure occurs includes:

[0029] When a loading failure occurs, the target log is written to the file system of the loading component of the destination server where the loading failure occurred, and a local file is obtained.

[0030] When the loading failure is recovered, depending on whether the transaction ID of the target log in the local file is the same as the release ID obtained from the concurrent conflict detection, the target log corresponding to the transaction ID can be selectively directly entered into the database, or the transaction corresponding to the transaction ID can be stored in the conflict transaction queue before the target log corresponding to the transaction ID is entered into the database.

[0031] Furthermore, when the loading failure is recovered, the step of selectively directly loading the target log corresponding to the transaction ID into the database, or storing the transaction corresponding to the transaction ID in the conflict transaction queue before loading the target log corresponding to the transaction ID into the database, based on whether the transaction ID of the target log in the local file is the same as the release ID obtained from the concurrent conflict detection, includes:

[0032] If the transaction ID is different from the ID to be released, the target log corresponding to the transaction ID is directly entered into the database and sent to the destination server.

[0033] If the transaction ID is the same as the ID to be released, then when the transaction corresponding to the transaction ID has been entered into the database, the target log corresponding to the transaction ID is directly entered into the database and sent to the destination server; when the transaction corresponding to the transaction ID has not been entered into the database, the transaction corresponding to the transaction ID is stored in the conflict transaction queue.

[0034] After each transaction is successfully written to the destination server, starting from the first transaction in the conflict transaction queue, it is checked in turn whether there is a transaction carrying a pending release ID that is the same as the transaction ID of the currently successfully written transaction.

[0035] Furthermore, the transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; the transformation of the transaction log through at least one transformation component to obtain the target log includes:

[0036] After obtaining the transaction log through the transformation dispatch thread, query the transformation mapping structure to see if the transaction corresponding to the transaction log has been assigned to an execution transformation thread.

[0037] If not allocated, an execution transformation thread is allocated to the transaction through the transformation distribution thread, and the transaction and its corresponding execution transformation thread are written into the transformation mapping structure; wherein, after the transaction transmission is completed, the transaction is deleted from the transformation mapping structure;

[0038] If already allocated, the transaction log is passed to the corresponding execution conversion thread via the conversion dispatch thread;

[0039] The allocated transaction logs are processed by the conversion thread. When the transaction logs match the user conversion rules, the corresponding conversion operations are performed on the transaction logs to obtain the target logs.

[0040] Furthermore, the loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; the step of loading the target log into the database through the loading component includes:

[0041] The target log is obtained through the loading and distribution thread, and the database connection object corresponding to the target log is queried in the loading mapping structure.

[0042] If no connection is allocated, when an idle database connection object exists, the idle database connection object is allocated to the transaction through the load dispatch thread and recorded in the load mapping structure; the idle database connection object is placed in the load ready queue, and the target log is retrieved again until no target log sent by the transformation component is available.

[0043] If already allocated, the target log is transmitted to the corresponding database connection object via the loading and distribution thread; by executing the loading thread, the database connection object generates an insert command based on the target log, and executes the insert command to insert the target log into the database.

[0044] Further, combining at least one reading component and at least one loading component to obtain the desired data synchronization task; or, combining at least one reading component, at least one conversion component, and at least one loading component to obtain the desired data synchronization task includes:

[0045] The connection relationship between at least one read component and at least one load component is determined based on the processing logic of the transaction event log; or, the connection relationship between at least one read component, at least one transformation component, and at least one load component is determined based on the processing logic of the transaction log events.

[0046] Based on the connection relationship, the corresponding components are connected together using directed arrow lines to obtain the data synchronization task;

[0047] Among them, a reading component can be connected to at least one conversion component and / or at least one loading component, a conversion component can be connected to at least one conversion component and / or at least one loading component, and at least one of the loading components can be connected to the destination database.

[0048] Secondly, the present invention also provides a data synchronization and fault recovery apparatus for implementing the data synchronization and fault recovery method described in the first aspect, the apparatus comprising:

[0049] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the data synchronization and fault recovery method described in the first aspect.

[0050] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the data synchronization and fault recovery method described in the first aspect.

[0051] Unlike existing technologies, the present invention has at least the following beneficial effects:

[0052] This invention combines a reading component, a transformation component, and a loading component to obtain the required data synchronization task. It uses functional components to graphically implement complex data synchronization tasks, improving the scalability of the data synchronization system. The reading component acquires the transaction logs involved in the data synchronization task and transmits them to the transformation component based on the concurrency conflict detection results. The transformation component transforms the transaction logs to obtain the target log, which is then transmitted to the loading component. The loading component stores the target log in the database. When a loading failure occurs, the target log is recovered based on the concurrency conflict detection results. This invention establishes a transaction synchronization process that includes data transformation, supports rich data transformation operations, enhances the scalability of the data synchronization system, implements concurrency conflict detection for transactions, and optimizes the loading failure recovery strategy. It solves the problems of existing data synchronization systems based on incremental transaction logs, which suffer from low usability due to attribute file configuration for data transformation operations, poor performance due to serial execution methods, and data loss caused by stopping the entire data synchronization task when a loading failure occurs. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram illustrating a formal definition of a DML operation provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram illustrating a formal definition of a DDL operation provided in an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram illustrating a formal definition of a transaction provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the architecture of an incremental data synchronization system based on transaction logs provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of a data synchronization task provided in an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of a one-to-one loading fault provided by an embodiment of the present invention;

[0060] Figure 7 This is a schematic diagram of a one-to-many loading failure provided in an embodiment of the present invention;

[0061] Figure 8 This is a schematic diagram of the overall process of a data synchronization and fault recovery method provided in an embodiment of the present invention;

[0062] Figure 9 This is a schematic diagram of the internal design and workflow of a reading component provided in an embodiment of the present invention;

[0063] Figure 10 This is a schematic diagram of a specific process for step 10 of an embodiment of the present invention;

[0064] Figure 11 This is a schematic diagram of a specific process for step 20 in an embodiment of the present invention;

[0065] Figure 12 This is a schematic diagram of the internal design and workflow of another reading component provided in an embodiment of the present invention;

[0066] Figure 13 This is a schematic diagram of a specific process for step 203 in an embodiment of the present invention;

[0067] Figure 14 This is a schematic diagram of a specific process for step 204 in an embodiment of the present invention;

[0068] Figure 15 This is a schematic diagram of a specific process for step 30 in an embodiment of the present invention;

[0069] Figure 16 This is a schematic diagram of a specific process for step 40 in an embodiment of the present invention;

[0070] Figure 17 This is another specific flowchart of step 40 in an embodiment of the present invention;

[0071] Figure 18 This is a schematic diagram of another one-to-one loading failure provided by an embodiment of the present invention;

[0072] Figure 19 This is a schematic diagram of another one-to-many loading failure provided in an embodiment of the present invention;

[0073] Figure 20This is a schematic diagram of a specific process for step 402b in an embodiment of the present invention;

[0074] Figure 21 This is a schematic diagram of the architecture of a data synchronization and fault recovery device provided in an embodiment of the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0076] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0077] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0078] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium.

[0079] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0080] Example 1:

[0081] To better illustrate the data synchronization and fault recovery methods of this invention, the relevant concepts are explained below.

[0082] A transaction is the basic unit of database operations. A transaction consists of one or more events. Specifically, one or more DDL events constitute a transaction, and their execution takes effect immediately. One or more consecutive DML events also constitute a transaction, and each execution must ultimately be explicitly committed with a Commit or Rollback event to mark the completion of the transaction, the persistence of data, or the rollback of executed actions. Before persisting the data changes made by a transaction to the data files, the database management system writes the event information of the changed data to a log file (redo / undo log) in an append-only manner.

[0083] Concurrency is a fundamental feature supported by modern database management systems. Concurrency refers to the ability of multiple database users to perform read and write operations simultaneously, that is, multiple transactions are being executed at the same time, and the events of multiple transactions are being written to the log file at the same time.

[0084] Before describing the data synchronization method of this invention, we first define the transaction event log in a formal way: the transaction event log refers to the log events (Events) contained in the transaction. This invention considers two types of transaction event logs: DML events and DDL events.

[0085] like Figure 1 The figure shows a formal definition of a DML event according to an embodiment of the present invention; the content of a DML event also includes a rowID, a unique identifier of its data in the source database; in an optional embodiment, a DML log event can be represented as:<trxId,seqNo,lsn,X,beforeValue,afterValue> .

[0086] Abstractly speaking, DML operation log events only include two types: reading data and writing data. Since this embodiment of the invention only involves data synchronization, we only need to focus on the transaction event logs of data modification operations, i.e., data writing events, which are specifically divided into three event types: adding data, modifying data, and deleting data. All valid DML transactions end with a commit event.

[0087] like Figure 2 The diagram shows the formal definition of a DDL log event; a DDL operation log event, where one SQL statement corresponds to one transaction, and one DDL command script constitutes one DDL transaction, meaning one DDL event is one transaction. In an optional embodiment, a DDL log event can be represented as:<trxId,lsn,opScript> .

[0088] like Figure 3 The figure shows the formal definition of a transaction, which consists of an ordered set of events.

[0089] In the log file, transaction event logs from different transactions are interleaved. Transaction event logs from the same transaction are written to the log file in the order they were executed. The database management system does not guarantee that transaction event logs belonging to the same transaction are stored adjacently in the log file. Overall, the log file contains the order in which transactions were executed and committed, as well as the execution order of transaction event logs within a transaction. Therefore, when parsing the log file, the transaction event logs obtained in sequence will most likely belong to different transactions.

[0090] like Figure 4 As shown, the architecture of an incremental data synchronization system based on transaction logs generally includes two service components (servers): a source end and a destination end. The source end obtains transaction event logs and sends them to the destination end. The destination end converts the transaction event logs into SQL or other commands and submits them for execution, thereby achieving consistency with the data in the source database and completing data synchronization. The transaction server (TrxServer), marked with a dashed box, can, in optional embodiments, be an API provided by the source database management system for obtaining transaction logs, or it can be a function implemented by the source service component. The source service component parses the log file to obtain the transaction event logs. The transaction server can also integrate a message server, storing the parsed log files on the message server for multiple source database source servers or other programs to read.

[0091] like Figure 4 The transaction log incremental data synchronization system shown is used in master-slave database clusters, and its architecture is mostly a master-slave architecture with one master and one slave or one master and multiple slaves. The master-slave architecture is designed to solve the high availability problem of data service systems. If the master system fails, the slave system can immediately take over and provide data services to external applications, requiring complete data consistency between the master and slave databases. Master-slave data synchronization is a typical one-to-one or one-to-many data synchronization application, where the master and slave databases have completely identical data.

[0092] Since many commercial closed-source databases often require data transformation during data synchronization, the data in the source database (i.e., the data generation database) cannot be synchronized to the destination database unchanged. Therefore, it is necessary to transform the data in the source database in a certain way before synchronizing it to the destination database.

[0093] Existing technologies generally do not support data transformation operations, or can only achieve certain forms of data transformation through complex attribute file configurations. This results in poor usability and scalability, and a lack of rich and easy-to-use data transformation operations. Furthermore, the deployment of servers used for data transformation is complex. For example, deploying a one-to-many data synchronization task requires configuring multiple independent destination servers, significantly increasing the complexity and difficulty of monitoring the data synchronization system. This hinders fault monitoring, recovery, and maintenance of the data synchronization system, resulting in poor practicality. For complex data synchronization tasks, existing technologies lack independent and flexible design support; they design data synchronization tasks through configuration files or abstract data transformation function interfaces, lacking a visually appealing graphical design approach, making them inflexible and unintuitive.

[0094] like Figure 5 and Figure 6 As shown, when using a component-based data synchronization system, a data loading failure refers to a situation where, during the operation of the loading component, the destination server where the destination database is located crashes, or communication abnormalities caused by network congestion cause the loading component to be unable to load the transaction event logs into the destination server.

[0095] One simple approach to handling loading failures is to halt the entire synchronization task. Traditional Extract-Transform-Load (ETL) systems, which rely on direct data read-write operations, typically employ this method. In traditional ETL, the synchronized data objects are persisted to the source server. However, log files on the source server are generally only retained for a limited time; they are truncated when the timeout period expires or the set size is reached. Data synchronization systems based on transaction event logs may encounter log file truncation if they fail to parse log files and read event information promptly, leading to data loss.

[0096] like Figure 7 As shown, in a one-to-many data synchronization task, an anomaly in the loading of a single loading component can affect the loading process of other loading components. Simply stopping the entire synchronization task will affect normally loading components; simply stopping the malfunctioning loading component will lead to inconsistencies between the corresponding destination database and other destination databases. Furthermore, a mechanism must be implemented to detect the faulty destination server or network recovery and restart the loading component. In addition, it increases the management load on the read component's data cache, as the read component needs to be notified to clear and release the cache after the transaction loading is completed.

[0097] To solve the aforementioned problems, such as Figure 8 As shown, this embodiment of the invention provides a method for data synchronization and fault recovery, including:

[0098] Step 10: Combine at least one read component and at least one load component to obtain the required data synchronization task; or, combine at least one read component, at least one transformation component and at least one load component to obtain the required data synchronization task.

[0099] This invention employs functional components to design data synchronization tasks. A functional component is a component that provides a single function, and includes three types: reading components, transformation components, and loading components. Specifically, the reading component acquires transaction event logs, the transformation component transforms the transaction event logs, and the loading component applies the transaction event logs to the destination database for execution.

[0100] This invention obtains the required data synchronization task by selecting at least one reading component and at least one loading component, and combining them with a conversion component selected from the two according to usage requirements. It also supports users designing data synchronization tasks by dragging and dropping functional graphical components. Logically, transaction event logs flow and are processed between various functional components, enabling support for complex data synchronization tasks. Using the component-based data synchronization method of this invention, data synchronization can be achieved without complex attribute file configuration operations for the reading, conversion, and loading components.

[0101] Based on the requirements of the data synchronization task, no conversion component is needed between the read component and the load component in some or all branches of the data synchronization task process. In the scenario of primary-standby architecture (one primary and one standby or one primary and multiple standby), the standby data is completely consistent with the primary data, and no data conversion is required. Therefore, there is no need to use a conversion component, and the read component and load component of the present invention can be directly combined to achieve data synchronization.

[0102] Step 20: Obtain the transaction logs involved in the data synchronization task through the reading component; selectively release conflicting transactions or detect concurrent conflicts in the transaction logs based on whether the transaction logs have been stored in the database; transmit the transaction logs to all subsequent components based on the result of the conflicting transaction release or the result of the concurrent conflict detection; wherein, the subsequent components of the reading component are the transformation component and the loading component.

[0103] The transaction log is a complete log of transaction events contained in a transaction, that is, all transaction event logs included in a transaction log.

[0104] The reading component in this embodiment of the invention collects transaction logs for concurrent conflict detection, distributes the transaction logs (including releasing conflicting transactions), and cleans up the transaction logs (i.e., clears the target log from the cache). Figure 9As shown, when distributing each transaction event log in the transaction log to subsequent components, a concurrency conflict check must be performed with the previously distributed and currently executing transactions. If no conflict exists, the transaction event logs without conflict are submitted to the subsequent components; if a conflict exists, the transaction to which the conflicting transaction event logs belong is mounted. Each time the synchronous execution of a transaction is completed, the reading component is responsible for clearing all transaction event log data for that transaction from the cache.

[0105] Specifically, for small transactions, the transaction logs involved in the data synchronization task are read by reading all transaction event logs issued by the component each time the transaction is processed, and in an optional embodiment, they are transmitted in the form of transaction event log groups; for transactions containing large fields or long transactions, the transaction event logs of a portion of the transaction being executed each time the component is processed are read.

[0106] The transaction logs are retrieved from the transaction server, cached, and distributed to subsequent components. During this process, a concurrency conflict check is performed between the logs and previously distributed and currently executing transactions. If no conflict exists, the transaction logs are committed to the subsequent component; otherwise, the transaction to which the conflicting logs belong is mounted. Furthermore, each time a transaction's log is written to the database, the reading component is responsible for clearing the data from the cache.

[0107] Step 30: When the subsequent component of the reading component is a transformation component, the transaction log is transformed by at least one transformation component to obtain the target log, and the target log is transmitted to the loading component; the target log is then stored in the database by the loading component.

[0108] In a data synchronization task, multiple transformation components can be cascaded. Each transformation component operates internally with identical logic; the only differences lie in the data source and data flow. When only one transformation component exists between the read and load components, its data source is the read component, and it transmits the transformed target log to the load component. When more than one transformation component exists between the read and load components, for a transformation component connected to the read component, its data source is the read component, and it transmits the transformed target log to the next transformation component connected to it; for a transformation component connected to a transformation component, its data source is the transformation component, and it transmits the further transformed target log to the next transformation component connected to it or the load component connected to it.

[0109] In this embodiment of the invention, each transformation component transforms and distributes the transaction logs. It obtains the transaction logs from the upstream component (i.e., the reading component or the previous transformation component with which it is connected) and performs data transformation operations on the transaction log data according to user-defined rules and methods. After the transformation is complete, the transformed target log is submitted to the subsequent component. The subsequent component can be either a transformation component or a loading component, depending on the specific task design of the data synchronization task.

[0110] Step 40: When the subsequent component of the reading component is the loading component, the target log is stored in the database through the loading component; when a loading failure occurs, the target log is restored according to the result of the concurrent conflict detection.

[0111] This invention designs a data synchronization system by combining componentization and database transaction logs. While integrating the advantages of both, it also introduces new challenges. In order to achieve the highest possible concurrency of multiple transactions, a loading failure handling process based on local caching is established under the premise of ensuring data synchronization consistency and correctness. Combined with transaction concurrency conflict detection, when the network or the destination server returns to normal, the cached transaction logs can be submitted to the destination server for execution as soon as possible to restore the target logs. This solves the problem of data loss caused by stopping the entire data synchronization task when a loading failure occurs in the prior art.

[0112] This invention combines a reading component, a transformation component, and a loading component to obtain the required data synchronization task. It uses functional components to graphically implement complex data synchronization tasks, improving the scalability of the data synchronization system. The reading component obtains the transaction logs involved in the data synchronization task and transmits them to the transformation component based on the concurrency conflict detection results. The transformation component transforms the transaction logs to obtain the target log, which is then transmitted to the loading component. The loading component stores the target log in the database. When a loading failure occurs, the target log is recovered based on the concurrency conflict detection results. This invention establishes a transaction synchronization process that includes data transformation, supports rich data transformation operations, enhances the scalability of the data synchronization system, implements concurrency conflict detection for transactions, and optimizes the loading failure recovery strategy. It solves the problems of existing data synchronization systems based on incremental transaction logs, which suffer from low usability due to attribute file configuration for data transformation operations, poor performance due to serial execution methods, and data loss caused by stopping the entire data synchronization task when a loading failure occurs.

[0113] To address the issue of complex attribute file configuration, this invention provides a component-based incremental data synchronization method. Specifically, the functional components include a reading component, a transformation component, and a loading component, such as... Figure 10 As shown, step 10 includes:

[0114] Step 101: Determine the connection relationship between at least one read component and at least one load component based on the processing logic of the transaction event log; or, determine the connection relationship between at least one read component, at least one transformation component, and at least one load component based on the processing logic of the transaction log event.

[0115] This invention designs data synchronization tasks by combining different functional components. The data synchronization system processes data according to the user-designed data synchronization tasks; each functional component runs on an independent thread. Data is transferred between functional components using a synchronization queue. The data transferred between functional components consists of all transaction event logs or partially consecutively executed transaction event logs belonging to the same transaction. Data flowing between two functional components belongs to the same transaction; for small transactions, a single data flow between functional components includes all transaction event logs of that transaction; for transactions containing large fields or long transactions, it generally only includes partially consecutively executed transaction event logs of that transaction.

[0116] Step 102: Connect the corresponding components together with directed arrows according to the connection relationship to obtain the data synchronization task.

[0117] In this configuration, a read component can be connected to at least one transformation component and / or at least one load component, a transformation component can be connected to at least one transformation component and / or at least one load component, and at least one load component can be connected to the destination database. In an optional embodiment, the source database is connected to at least one read component, the transformation component is generally connected to only one read component or transformation component, and the load component is generally directly connected to the destination database (such as a central database or a relational database).

[0118] A component-based data synchronization system abstracts the data synchronization process into three basic steps: Extract, Transform, and Load, corresponding to three basic functional components: a extract component, a transform component, and a load component. For example... Figure 5 As shown, the component-based data synchronization system features a single, highly functional component designed for clarity and immediate usability. It supports drag-and-drop component design for data synchronization tasks, demonstrating exceptional user-friendliness. Users select different functional components as needed and combine them according to the data flow processing order to design data synchronization tasks. Unlike traditional ETL components that flow individual data records, the component-based data synchronization system of this invention facilitates the flow of transaction logs between components.

[0119] The internal operating logic of the data synchronization task is explained below:

[0120] During data synchronization, each functional component starts an independent execution thread. Transaction logs can be exchanged between functional components and their directly connected counterparts using message queues. A pipelined data processing process for the read component consists of a transaction acquisition thread, a conflict detection thread, a distribution thread, and a cleanup thread. Transaction event logs within the same transaction are processed by the same execution transformation thread, with multiple execution transformation threads concurrently performing data processing in the transformation component. Transactions are bound to database connection objects, and transaction event logs within the same transaction are submitted to the destination database through the same database connection object. Multiple transaction loading threads concurrently perform data processing in the loading component. Each read component, each transformation component, and each loading component runs using an independent execution thread. Based on the data processing processes of the read component, the transformation component, and the loading component, a data synchronization task is executed to complete the data synchronization.

[0121] The transaction logs parsed from the source database service system's redo / undo files inherently possess a conflict-free, sequential execution relationship. The reading component can distribute transaction logs according to this sequential order, waiting for the previous transaction to complete before distributing the logs for the next transaction. This execution method ensures the correctness of synchronized data but suffers from low execution efficiency. This invention provides a method for concurrent conflict detection using multiple transactions, enabling concurrent execution of multiple transactions in conflict-free situations to improve efficiency. A conflict refers to multiple transactions operating on the same data object and modifying it. If two transactions conflict, they must synchronize their data according to their commit times.

[0122] To further illustrate the internal process of reading the component, such as Figure 11 As shown, step 20 specifically includes:

[0123] Step 201: After obtaining the complete transaction log of the transaction to be detected, the transaction to be detected is set as the last transaction in the complete transaction sequence linked list, and the transaction to be detected is set as the last ready transaction in the transmission ready queue.

[0124] Among them, the transaction to be detected refers to the transaction log obtained by the reading component from the source database.

[0125] like Figure 9 As shown, the functional modules of the reading component are divided into "transaction acquisition and caching module", "transaction detection and concurrency module" and "cleanup module"; in an optional embodiment, the complete transaction sequence linked list can be implemented by an ordered linked list or a skip list.

[0126] like Figure 12 As shown, the complete transaction sequence list organizes transactions according to the order of transaction log collection (which is also the commit order of transactions on the source database server, with transactions committed earlier listed first). This includes all transactions blocked due to conflicts and transactions that have been issued and are being processed. Completed transactions need to be removed from the complete transaction sequence list in a timely manner, which is done by the transaction cleanup module in step (6.2). After collecting the complete transaction to be detected, the event collection and caching module synchronously executes steps (2.1) and (2.2) respectively, attaching the transaction to be detected to the end of the transmission ready queue and the complete transaction sequence list.

[0127] Step 202: Obtain the completed transaction from the loading completion notification queue, remove the completed transaction from the complete transaction sequence list, and make the completed transaction the first ready transaction in the transmission ready queue; wherein, the complete transaction sequence list includes transactions being transmitted and transactions in the conflicting transaction queue.

[0128] Among them, a completed transaction refers to a transaction in which the loading component has executed the data entry into the destination database.

[0129] Step 203: Take out ready transactions sequentially from the transmission ready queue and determine whether the transaction log of the ready transaction has been entered into the database.

[0130] Step 204: When the retrieved ready transaction is a completed transaction, determine the transaction that has a concurrent conflict with the ready transaction from the conflict transaction queue, and release the conflict transaction for the completed transaction; transmit the transaction log of the transaction that has a concurrent conflict with the ready transaction to all subsequent components.

[0131] Step 205: When the retrieved ready transaction is not a completed transaction, determine whether the ready transaction has a concurrent conflict with the transactions in the complete transaction sequence list, so as to perform concurrent conflict detection on the ready transaction; based on the result of the concurrent conflict detection, selectively attach the ready transaction to the conflict transaction queue, or transmit the transaction log of the ready transaction to all subsequent components.

[0132] In an optional embodiment, the transaction collection and caching module is responsible for collecting and caching transaction logs. After all transaction event logs for a transaction have been collected, they are submitted to the end of the transmission ready queue, and are mounted to the transmission ready queue in the order in which the transactions were committed in the source database.

[0133] The cleanup module retrieves notifications from the loading completion notification queue indicating that a transaction has been completed. It counts the number of completion notifications, and once all relevant loading components have finished processing, the transaction is considered complete and can be removed from the cache. The module then notifies the transaction detection and concurrency module that it can release any conflicting transactions. The cleanup module also promptly notifies the transaction detection and concurrency module to prioritize the release of transactions blocked in the conflicting transaction queue by adding completed transactions to the head of the transmission ready queue.

[0134] The transaction detection and concurrency module is responsible for detecting concurrency conflicts of ready transactions. When a concurrency conflict exists, the ready transaction is added to the conflict transaction queue. After the conflicting transaction is completed, it is submitted to the subsequent component for processing. When there is no concurrency conflict, the ready transaction is sent to the subsequent component and added to the transaction execution queue.

[0135] The event acquisition and caching module and the transaction cleanup module are responsible for providing operation objects to the transaction detection and concurrency module. The operation objects are ready transactions and completed transactions, which are submitted to the tail and head of the ready event queue, respectively. Completed transactions are processed first, and transactions that conflict with completed transactions in the conflict transaction queue are released as early as possible.

[0136] Operational details not mentioned in the embodiments of this invention shall be selected by those skilled in the art based on the specific application scenario and without creative effort, and shall not be elaborated here.

[0137] When a ready transaction in the transmission ready queue is not a completed transaction, to further explain the concurrent conflict detection, such as... Figure 13 As shown, step 203 includes:

[0138] Step 2031: Starting from the last transaction in the complete transaction sequence list, compare the transactions in the complete transaction sequence list with the current ready transaction in reverse order to determine whether there is any intersection between the data object operated by the current ready transaction and the data object operated by the transaction in the complete transaction sequence list.

[0139] Each event in a transaction corresponds to a modification of a database object. Assuming transaction i (Trx(i)) has k events, the corresponding set of data objects is denoted as {x(i1), x(i2), ..., x(ik)}. Concurrency conflict detection aims to determine whether the currently ready transaction, previously issued and currently being processed transactions, and conflicting blocked transactions have any overlap in the data objects they are operating on.

[0140] Step 2032: When there is an intersection, stop judging the current ready transaction with the transactions that have not been compared in the complete transaction sequence list, record the transaction ID of the transaction that intersects with the current ready transaction as the release ID, and attach the current ready transaction to the conflict transaction queue by carrying the release ID.

[0141] This involves adding new information to the blocked ready transactions, namely the transaction ID that blocked the ready transaction in the complete transaction sequence list.

[0142] Step 2033: When there is no intersection, the current ready transaction is stored in the execution transaction queue, so that the transaction log of the current ready transaction is transmitted to all subsequent components according to the execution transaction queue.

[0143] The transaction detection and concurrency module reads transaction objects from the ready transaction queue, such as... Figure 12 In step (7) shown, the transaction object is divided into ready transactions and completed transactions, which are submitted to the tail and head of the transmission ready queue by the event acquisition and caching module and the cleanup module, respectively.

[0144] The transaction detection and concurrency module of this invention processes transactions read from the transmission ready queue according to the two methods described above. Conflicting transactions are released and sent out. The transaction IDs of blocked transactions in the complete transaction sequence list are compared with the IDs of completed transactions to determine whether the blocked transactions can be safely released and sent out (i.e., there are no concurrent conflicts). This information will also be used in subsequent load failure recovery.

[0145] When a ready transaction in the ready queue is a completed transaction, in order to allow ready transactions that have concurrency conflicts with completed transactions to be executed, such as... Figure 14 As shown, step 204 includes:

[0146] Step 2041: Starting from the first transaction in the conflict transaction queue, compare the pending release ID carried by each transaction in the conflict transaction queue with the transaction ID of the completed transaction in turn to determine whether the completed transaction causes a conflict blockage to the transactions in the conflict transaction queue.

[0147] Step 2042: When the ID to be released is the same as the transaction ID of the completed transaction, the transaction log corresponding to the ID to be released is transmitted to all subsequent components, and the ID to be released is stored in the execution transaction queue.

[0148] Iterate through the transaction at the head of the conflict transaction queue, comparing the transaction ID that caused the conflict with the transaction ID of the currently completed transaction. If they are the same, it means that the conflict factor of the transaction has been resolved and it can be released and issued. After issuance, its transaction ID is then added to the execution transaction queue.

[0149] Step 2043: After all transactions in the conflict transaction queue have been evaluated, the completed transaction is removed from the execution transaction queue.

[0150] During concurrent conflict detection, new information is added to ready transactions that need to be blocked: the transaction ID of the blocking transaction is carried in the ordered linked list of complete transactions. When releasing conflicting transactions from completed transactions, the carried transaction ID is compared with the transaction ID of the completed transaction to find the transaction blocked by the completed transaction in the conflicting transaction queue and safely release it.

[0151] To further illustrate the internal process of the conversion component, such as Figure 15 As shown, the transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; in step 30, the transformation of the transaction log through at least one transformation component to obtain the target log includes:

[0152] Step 301: After obtaining the transaction log through the transformation dispatch thread, query the transformation mapping structure to see if the transaction corresponding to the transaction log has been assigned to an execution transformation thread.

[0153] The transformation component transforms the data in the transaction log. A transaction log includes at least one transaction event log, and each transaction event log corresponds to a change to a data object. For relational databases, a transaction event log corresponds to a change to a row of data, typically including the schema name, table name, array of column names, array of new column values, and array of old column values. The transformation component transforms the above data content in the transaction event log, such as replacing the value of a column in the array of new column values, renaming a column in the array of column names, and modifying the table name and schema name.

[0154] The conversion component is internally designed with multiple worker threads. Each conversion thread processes the transaction log of one transaction to ensure the order of transaction event logs within the same transaction. After processing, the target log after the corresponding transaction log conversion is immediately sent out.

[0155] Step 302: If not allocated, an execution conversion thread is allocated to the transaction through the conversion distribution thread, and the transaction and the corresponding execution conversion thread are written into the conversion mapping structure; wherein, after the transaction is transmitted, the transaction is deleted from the conversion mapping structure.

[0156] Step 303: If allocated, the transaction log is passed to the corresponding execution transformation thread via the transformation dispatch thread.

[0157] In this embodiment of the invention, after the transformation distribution thread obtains the transaction log issued by the reading component, it checks in the transformation mapping structure whether the transaction has been assigned to an execution transformation thread. If it has been assigned, the transaction log is passed to the execution transformation thread; if it has not been assigned, an execution transformation thread is selected according to the rules set by the user for the processing of the current transaction, and the transaction ID and the identifier of the execution transformation thread are written into the transformation mapping structure.

[0158] Step 304: Process the allocated transaction logs by executing the transformation thread. When the transaction logs match the user's transformation rules, perform the corresponding transformation operation on the transaction logs to obtain the target logs.

[0159] When the transaction log matches the user-defined transformation rules, the corresponding transformation operation is performed on the data in the transaction log; otherwise, no transformation is required. After the transaction log is processed, it is sent to subsequent components. After processing the transaction log of a transaction (i.e., transforming the transaction event log of the commit event), the transformation thread removes the mapping item for that transaction from the transformation mapping structure.

[0160] To illustrate the loading process of the loading components, such as Figure 16 As shown, specifically, the loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; in steps 30 and 40, the step of loading the target log into the database through the loading component includes:

[0161] Step 401a: Obtain the target log through the load distribution thread, and query in the load mapping structure whether the transaction corresponding to the target log has been allocated a database connection object.

[0162] The loading component maintains a database connection pool and a loading mapping structure; a transaction allocates a database connection object from the database connection pool, and the loading mapping structure is used to store the mapping relationship between the database connection object and the transaction; a transaction's transaction log (all transaction event logs) is submitted to the destination database for execution via the database connection object.

[0163] Step 402a: If not allocated, when an idle database connection object exists, the idle database connection object is allocated to the transaction through the load distribution thread and recorded in the load mapping structure; the idle database connection object is placed in the load ready queue, and the target log is retrieved again until no target log sent by the transformation component is available.

[0164] Step 403a: If already allocated, the target log is transmitted to the corresponding database connection object via the loading and distribution thread; by executing the loading thread, the database connection object generates an insert command based on the target log and executes the insert command to insert the target log into the database.

[0165] The loading component directly or indirectly submits the transaction log to the destination database for insertion. The loading thread retrieves a database connection object from the ready queue and executes the transaction log within that connection object. Each time a database connection object is retrieved from the ready queue, insertion commands are submitted to the destination database one by one according to the order of the transaction event logs within the transaction, and the transaction is inserted according to the corresponding insertion commands. If the waiting connection queue is empty, the database connection object is returned to the database connection pool. The loading thread continues to retrieve database connection objects from the ready queue, executing the transaction assigned to the database connection object upon retrieval; otherwise, it waits.

[0166] In an optional embodiment, for a relational database, a transaction event log can generate an SQL statement. The SQL statement is submitted to the destination database for execution. All transaction event logs of the same transaction generate SQL statements in sequence and submit them to the destination database for execution. Then, a commit command is submitted. Only after successful execution is the synchronization of a transaction considered complete.

[0167] After a transaction is completed (committed to the database), the database connection object and the transaction allocation relationship are released. The corresponding read component is then notified of the transaction ID of the completed transaction. The read component then distributes ready transactions that have concurrency conflicts with the completed transaction from the conflict transaction queue based on the transaction ID. After that, the transaction log of the completed transaction is cleared from the cache. If the completed transaction is eventually distributed to multiple loading components, the transaction log of the completed transaction can only be cleared from the cache after all loading components have finished executing.

[0168] To further illustrate the loading failure recovery strategy of this invention, such as Figure 17 As shown, specifically, in step 40, restoring the target log based on the result of the concurrent conflict detection when a loading failure occurs includes:

[0169] Step 401b: When a loading failure occurs, the target log is written to the file system of the loading component of the destination server where the loading failure occurred, and a local file is obtained.

[0170] A loading failure prevents communication between the loading component and the destination server. Possible causes include the target log server crashing due to an unexpected failure, the destination server being unresponsive due to excessive load, or network congestion preventing normal access to the destination server. When such an anomaly occurs, this embodiment of the invention employs a basic strategy of writing to a local file and recovering based on that file to achieve fault recovery. When the destination server is inaccessible, the transaction log is written to the file system of the machine where the loading component resides.

[0171] Step 402b: When the loading failure is recovered, depending on whether the transaction ID of the target log in the local file is the same as the release ID obtained by the concurrent conflict detection, the target log corresponding to the transaction ID can be selectively directly entered into the database, or the transaction corresponding to the transaction ID can be stored in the conflict transaction queue and then the target log corresponding to the transaction ID can be entered into the database.

[0172] In an optional embodiment, a thread is simultaneously started to probe whether the remote data server (i.e., the destination server) can be accessed normally. If the remote data server is detected to have recovered, a new remote loading thread is started to read the transaction log from the local file and then submit it to the remote data server. Once the transaction log in the local file has been read and processed, the fault recovery is completed, and the system switches back to normal working mode.

[0173] It is worth noting that the transaction logs after being transformed by the transformation component are written to local files in the form of appending. After the destination server resumes normal access, the loading component starts a new loading thread to process the transaction logs cached in local files in a sequential manner and submits them to the destination server in the order they were written to the files. In a serial submission manner, the next transaction is submitted only after the previous transaction has been completed, so as to achieve the purpose of data synchronization and ensure data consistency.

[0174] After the loading component writes the transaction log to a local file, it still notifies the reading component that the transaction has been processed according to the normal workflow. The "transaction cleanup module" of the reading component continues to process the transaction according to the normal workflow. This fault handling method of writing to a local file will not affect the normal operation of other functional components.

[0175] The advantage of the serial submission method is its simple control, but its disadvantage is its low performance. Figure 18 As shown, in one-to-one data synchronization tasks, apart from a slightly slower data synchronization speed, there are generally no other major impacts. However, in one-to-many data synchronization tasks, such as... Figure 19As shown, data from the source database is synchronized in real-time to three destination databases on dedicated servers: one for real-time data backup, one for full-text search services, and one for message subscription. When the message subscription service recovers from a crash, data loading should commit the locally cached transaction logs to the message server as quickly as possible to achieve data consistency with the other destination servers. If data inconsistency persists across the three destination servers, inconsistent data read by external applications from different destination servers may lead to unknown data application risks. The likelihood of this risk increases with the duration of data inconsistency.

[0176] To further illustrate the loading and recovery process, such as Figure 20 As shown, specifically, step 402b includes:

[0177] Step 4021: If the transaction ID is different from the ID to be released, then directly store the target log corresponding to the transaction ID into the destination server.

[0178] This invention aims to minimize the duration of data inconsistency across different destination servers. After recovery from a loading failure, a parallel commit execution method is used for recovery. Concurrent execution presents the same problem as the read component: transaction concurrency conflicts. While the concurrency conflict handling method in the read component can be simplified and directly applied to transaction concurrency processing here, overall, two conflict comparisons are performed on transactions cached in local files, resulting in redundant computational consumption. Looking back at the blocked transactions in the conflict transaction queue of the read component, each carries an additional piece of information: the pending release ID of the conflicting transaction. Once this transaction is entered into the database, all blocked transactions storing that transaction ID can be directly released. After being released, the blocked transactions in the read component must still carry the pending release ID. At this point, the concurrent transaction processing after the loading failure recovery can be reused, reducing the computational consumption of comparing the intersection of the data object sets operated on by the transactions.

[0179] Step 4022: If the transaction ID is the same as the ID to be released, then when the transaction corresponding to the transaction ID has been entered into the database, the target log corresponding to the transaction ID is directly entered into the destination server; when the transaction corresponding to the transaction ID has not been entered into the database, the transaction corresponding to the transaction ID is stored in the conflict transaction queue.

[0180] In an optional embodiment, a set of transaction IDs of executed inbound transactions is maintained; if a transaction in the conflict transaction queue does not carry an ID to be released, it is directly submitted to the remote server for inbound execution.

[0181] If a transaction in the conflict transaction queue carries a pending release ID, then the transaction ID set of the cached executed transactions is checked to see if the transaction has already been executed. If it has been executed, the current transaction can also be directly committed to execute the transaction. Otherwise, the current transaction is blocked and suspended in the conflict transaction queue until there are no more concurrent conflicts before it is committed to execute the transaction.

[0182] Step 4023: After each transaction is successfully stored in the destination server, starting from the first transaction in the conflict transaction queue, check sequentially from front to back whether there is a transaction carrying a pending release ID that is the same as the transaction ID of the currently successfully stored transaction.

[0183] In an optional embodiment, after a committed transaction is successfully written to the database, the blocked transactions are traversed starting from the head of the conflicting transaction queue. If the ID of the blocked transaction is the same as the ID of the currently executed and written-to-database transaction, the blocked transaction can be committed. At the same time, the ID of the currently executed and written-to-database transaction is added to the transaction ID set of executed transactions.

[0184] This invention combines concurrent conflict detection of the reading component with fault recovery of the loading component by using the pending release ID of conflicting transactions. This cleverly extends the calculation effect of concurrent conflict detection of transactions performed in the reading component to the concurrent conflict handling stage of the loading fault recovery, thereby improving the efficiency of loading fault recovery.

[0185] Example 2:

[0186] like Figure 21 The diagram shown is an architectural schematic of a data synchronization and fault recovery device according to an embodiment of the present invention. The data synchronization and fault recovery device of this embodiment includes one or more processors 31 and a memory 32. Figure 21 Take a processor 31 as an example.

[0187] Processor 31 and memory 32 can be connected via a bus or other means. Figure 21 Taking the example of a connection between China and Israel via a bus.

[0188] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the data synchronization and fault recovery method in Embodiment 1. The processor 31 executes the data synchronization and fault recovery method by running the non-volatile software program and instructions stored in the memory 32.

[0189] Memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory remotely located relative to processor 31, which can be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0190] The program instructions / modules are stored in the memory 32. When executed by one or more processors 31, they perform the data synchronization and fault recovery method in Embodiment 1 above, for example, performing each step shown in the data synchronization and fault recovery method described above.

[0191] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0192] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0193] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for data synchronization and fault recovery, characterized in that, include: The required data synchronization task is obtained by combining at least one read component and at least one load component. Alternatively, at least one reading component, at least one transformation component, and at least one loading component can be combined to obtain the required data synchronization task; The transaction logs involved in the data synchronization task are obtained by reading the component. Depending on whether the transaction logs have been entered into the database, conflicting transactions are released or concurrent conflicts are detected selectively. Based on the result of the conflicting transaction release or the result of the concurrent conflict detection, the transaction log is transmitted to all subsequent components, including: after obtaining the complete transaction log of the transaction to be detected, the transaction to be detected is designated as the last transaction in the complete transaction sequence list and the last ready transaction in the transmission ready queue; the completed transaction is removed from the complete transaction sequence list and designated as the first ready transaction in the transmission ready queue; ready transactions are sequentially retrieved from the transmission ready queue, and if it is a completed transaction, the transaction log of the conflicting transaction is determined from the conflicting transaction queue and transmitted to all subsequent components; if it is not a completed transaction, its concurrent conflict with the transactions in the complete transaction sequence list is detected, and the ready transaction is selected to be mounted to the conflicting transaction queue or transmitted to all subsequent components based on the detection result; wherein, the subsequent components of the reading component are the transformation component and the loading component. When the subsequent component of the reading component is a transformation component, the transaction log is transformed by at least one transformation component to obtain the target log, and the target log is transmitted to the loading component; the target log is then stored in the database by the loading component. When the subsequent component of the reading component is the loading component, the target log is loaded into the database through the loading component; when a loading failure occurs, the target log is restored according to the result of the concurrent conflict detection.

2. The data synchronization and fault recovery method according to claim 1, characterized in that, The process involves obtaining the transaction logs related to the data synchronization task through a reading component, and selectively releasing conflicting transactions or detecting concurrent conflicts based on whether the transaction logs have been stored in the database. Based on the result of the conflicted transaction release or the result of the concurrent conflict detection, transmitting the transaction log to all subsequent components includes: The completed transaction is obtained from the loading completion notification queue, removed from the complete transaction sequence list, and designated as the first ready transaction in the transmission ready queue; wherein, the complete transaction sequence list includes transactions being transmitted and transactions in the conflicting transaction queue; Ready transactions are retrieved sequentially from the transmission ready queue, and it is determined whether the transaction log of the ready transaction has been entered into the database. When the retrieved ready transaction is a completed transaction, the transaction that has a concurrent conflict with the ready transaction is identified from the conflict transaction queue, and the conflict transaction is released for the completed transaction; the transaction log of the transaction that has a concurrent conflict with the ready transaction is transmitted to all subsequent components. When a ready transaction is not a completed transaction, it is determined whether the ready transaction has a concurrent conflict with the transactions in the complete transaction sequence list, so as to perform concurrent conflict detection on the ready transaction; based on the result of the concurrent conflict detection, the ready transaction is selectively attached to the conflict transaction queue, or the transaction log of the ready transaction is transmitted to all subsequent components.

3. The data synchronization and fault recovery method according to claim 2, characterized in that, When the retrieved ready transaction is not a completed transaction, it is determined whether the ready transaction has a concurrent conflict with the transactions in the complete transaction sequential list, so as to perform concurrent conflict detection on the ready transaction; Based on the results of concurrent conflict detection, selectively attaching the ready transactions to the conflict transaction queue, or transmitting the transaction logs of the ready transactions to all subsequent components, includes: Starting from the last transaction in the complete transaction sequence list, compare the transactions in the complete transaction sequence list with the current ready transaction in reverse order to determine whether there is any intersection between the data object operated by the current ready transaction and the data object operated by the transactions in the complete transaction sequence list. When there is an intersection, stop judging the current ready transaction with the transactions that have not been compared in the complete transaction sequence list, record the transaction ID of the transaction that has an intersection with the current ready transaction as the ID to be released, make the current ready transaction carry the ID to be released, and attach the current ready transaction to the conflict transaction queue. When there is no intersection, the current ready transaction is stored in the execution transaction queue, so that the transaction log of the current ready transaction is transmitted to all subsequent components according to the execution transaction queue.

4. The data synchronization and fault recovery method according to claim 2, characterized in that, When the retrieved ready transaction is a completed transaction, the transaction that has a concurrent conflict with the ready transaction is determined from the conflict transaction queue, and the conflict transaction is released for the completed transaction. Transmitting the transaction logs of transactions that conflict with ready transactions to all subsequent components includes: Starting from the first transaction in the conflict transaction queue, the pending release ID carried by each transaction in the conflict transaction queue is compared with the transaction ID of the completed transaction in order to determine whether the completed transaction causes a conflict or blockage to the transactions in the conflict transaction queue. When the ID to be released is the same as the transaction ID of the completed transaction, the transaction log corresponding to the ID to be released is transmitted to all subsequent components, and the ID to be released is stored in the execution transaction queue. After all transactions in the conflict transaction queue have been evaluated, the completed transaction is removed from the execution transaction queue.

5. The data synchronization and fault recovery method according to claim 1, characterized in that, The step of restoring the target log based on the result of the concurrent conflict detection when a loading failure occurs includes: When a loading failure occurs, the target log is written to the file system of the loading component of the destination server where the loading failure occurred, and a local file is obtained. When the loading failure is recovered, depending on whether the transaction ID of the target log in the local file is the same as the release ID obtained from the concurrent conflict detection, the target log corresponding to the transaction ID can be selectively directly entered into the database, or the transaction corresponding to the transaction ID can be stored in the conflict transaction queue before the target log corresponding to the transaction ID is entered into the database.

6. The data synchronization and fault recovery method according to claim 5, characterized in that, When the loading failure is recovered, the step of selectively directly loading the target log corresponding to the transaction ID into the database, or storing the transaction corresponding to the transaction ID in the conflict transaction queue before loading the target log corresponding to the transaction ID into the database, based on whether the transaction ID of the target log in the local file is the same as the release ID obtained from the concurrent conflict detection, includes: If the transaction ID is different from the ID to be released, the target log corresponding to the transaction ID is directly entered into the database and sent to the destination server. If the transaction ID is the same as the ID to be released, then when the transaction corresponding to the transaction ID has been entered into the database, the target log corresponding to the transaction ID is directly entered into the database and sent to the destination server; when the transaction corresponding to the transaction ID has not been entered into the database, the transaction corresponding to the transaction ID is stored in the conflict transaction queue. After each transaction is successfully written to the destination server, starting from the first transaction in the conflict transaction queue, it is checked in turn whether there is a transaction carrying a pending release ID that is the same as the transaction ID of the currently successfully written transaction.

7. The data synchronization and fault recovery method according to claim 1, characterized in that, The transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; the transformation of the transaction log through at least one transformation component to obtain the target log includes: After obtaining the transaction log through the transformation dispatch thread, query the transformation mapping structure to see if the transaction corresponding to the transaction log has been assigned to an execution transformation thread. If not allocated, an execution transformation thread is allocated to the transaction through the transformation distribution thread, and the transaction and its corresponding execution transformation thread are written into the transformation mapping structure; wherein, after the transaction transmission is completed, the transaction is deleted from the transformation mapping structure; If already allocated, the transaction log is passed to the corresponding execution conversion thread via the conversion dispatch thread; The allocated transaction logs are processed by the conversion thread. When the transaction logs match the user conversion rules, the corresponding conversion operations are performed on the transaction logs to obtain the target logs.

8. The data synchronization and fault recovery method according to claim 1, characterized in that, The loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; the step of loading the target log into the database through the loading component includes: The target log is obtained through the loading and distribution thread, and the database connection object corresponding to the target log is queried in the loading mapping structure. If no connection is allocated, when an idle database connection object exists, the idle database connection object is allocated to the transaction through the load dispatch thread and recorded in the load mapping structure; the idle database connection object is placed in the load ready queue, and the target log is retrieved again until no target log sent by the transformation component is available. If already allocated, the target log is transmitted to the corresponding database connection object via the loading and distribution thread; by executing the loading thread, the database connection object generates an insert command based on the target log, and executes the insert command to insert the target log into the database.

9. The data synchronization and fault recovery method according to claim 1, characterized in that, The process involves combining at least one reading component and at least one loading component to obtain the required data synchronization task. Alternatively, combining at least one read component, at least one transformation component, and at least one load component yields the desired data synchronization task, including: The connection relationship between at least one read component and at least one load component is determined based on the processing logic of the transaction event log; or, the connection relationship between at least one read component, at least one transformation component, and at least one load component is determined based on the processing logic of the transaction log events. Based on the connection relationship, the corresponding components are connected together using directed arrow lines to obtain the data synchronization task; Among them, a reading component can be connected to at least one conversion component and / or at least one loading component, a conversion component can be connected to at least one conversion component and / or at least one loading component, and at least one of the loading components can be connected to the destination database.

10. A data synchronization and fault recovery device, characterized in that, It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. When executed by the processor, the instructions are used to implement the data synchronization and fault recovery method according to any one of claims 1-9.

Images