An assembly data synchronization method and apparatus

Abstract

The application relates to the technical field of data synchronization, and provides a componentized data synchronization method and device. The application combines a reading component, a conversion component and a loading component to obtain a required data synchronization task, uses functional components to realize a complex data synchronization task in a graphical manner, obtains transaction event logs related to the data synchronization task through the reading component, transmits the transaction event logs to the conversion component, converts the transaction event logs through the conversion component to obtain target data, and transmits the target data to the loading component; the target data is warehoused through the loading component; the application establishes a transaction synchronization process containing data conversion, supports rich data conversion operations, makes the data synchronization system highly extensible, solves the problems that the prior art realizes data conversion operations through attribute file configuration, is inconvenient to operate and maintain, has low usability, and task design is not very flexible.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a modular data synchronization method and apparatus. Background Technology

[0002] Database management systems (DBMS) use log files to store records of data changes. When events or actions are performed, including data modification operations such as Data Manipulation Language (DML) operations (writing, modifying, and deleting data) and Data Definition Language (DDL) operations (creating and modifying tables), the DBMS appends these changes to the log file. In the event of an unexpected DBMS crash, data can be recovered from the log file upon restart, ensuring data consistency. This was the original purpose of introducing log files into DBMSs. The log file contains all the details of data changes, which can be used for data synchronization, especially incremental data synchronization. Synchronizing data based on the log file fundamentally solves the challenges of incremental data synchronization.

[0003] Existing incremental data synchronization systems based on transaction logs are generally limited to a source-destination architecture, consisting of two service components: a source and a destination. The source component acquires transaction event logs and sends them to the destination. The destination component converts the transaction event logs into Structured Query Language (SQL) or other commands and submits them to the destination database server for execution, thereby achieving data consistency with the source database and completing data synchronization. This architecture is typically used to achieve fully consistent data synchronization tasks with one master and one backup or one master and multiple backups. It does not support data transformation operations or relies on complex attribute file configurations to achieve data synchronization and some form of data transformation, making it inconvenient to operate and maintain. Furthermore, existing technologies design data synchronization tasks through configuration files or abstract data transformation function interfaces, lacking independent and flexible design support for complex data synchronization tasks.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a modular data synchronization method and apparatus. Its purpose is to establish a transaction synchronization process that includes data transformation, supports rich data transformation operations, and designs complex data synchronization tasks in a graphical way through functional components. This makes the data synchronization system highly scalable and solves the problems of existing data synchronization systems based on transaction log incrementally, which implement data transformation operations through attribute file configuration, making them inconvenient to operate and maintain, with low usability and very inflexible task design.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a component-based data synchronization method, comprising:

[0008] The required data synchronization task can be obtained by combining at least one read component and at least one load component; or, the required data synchronization task can be obtained by combining at least one read component, at least one transformation component, and at least one load component.

[0009] The transaction event logs involved in the data synchronization task are obtained by the reading component, and the transaction event logs are transmitted to all subsequent components of the reading component; wherein, the subsequent components of the reading component are transformation components or loading components.

[0010] When the subsequent component of the reading component is a transformation component, the transaction event log is transformed by at least one transformation component to obtain the target data; the target data is then transferred to the loading component; and the target data is then stored in the database by the loading component.

[0011] When the subsequent component of the reading component is a loading component, the target data is loaded into the database through the loading component.

[0012] Furthermore, the step of obtaining the transaction event logs involved in the data synchronization task through the reading component and transmitting the transaction event logs to all subsequent components of the reading component includes:

[0013] Obtain transaction event logs belonging to the same transaction and store them in the corresponding preset cache set; wherein, the transaction event logs belonging to the same transaction are all transaction event logs or some consecutive transaction event logs of the transaction;

[0014] When the transaction event log is a commit event, the transaction event log corresponding to the commit event is added from the preset cache set to the transmission ready queue to obtain a ready transaction;

[0015] Depending on whether there are concurrent conflicts among the ready transactions in the ready queue, the corresponding ready transactions are selectively transmitted to all subsequent components; or the corresponding ready transactions are attached to the conflict transaction queue until the transactions that have concurrent conflicts with the ready transactions are completed, at which point the ready transactions are removed from the conflict transaction queue and transmitted to all subsequent components.

[0016] After the transaction event log of the current transaction is processed, the transaction event log of the next transaction is retrieved, until a stop retrieval instruction is received.

[0017] Further, the step of selectively transmitting the corresponding ready transaction to all subsequent components based on whether there are concurrent conflicts in the ready transaction queue; or attaching the corresponding ready transaction to the conflict transaction queue includes:

[0018] Sequentially determine whether there are any transaction event logs in the execution transaction queue that conflict with the current ready transaction; when there is a conflict, record all transaction event logs that conflict with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue.

[0019] Sequentially determine whether there are any transaction event logs in the conflict transaction queue that conflict with the current ready transaction; when there is a conflict, record all transaction event logs that conflict with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue.

[0020] When there are no concurrent conflicts in the current ready transaction, the current ready transaction is added to the execution transaction queue, and the ready transaction is transmitted to all subsequent components according to the execution transaction queue.

[0021] Furthermore, after the transaction event log of the current transaction is processed, the transaction event log of the next transaction is obtained, until a stop acquisition instruction is received, including:

[0022] Retrieve completed transactions from the loading completion notification queue, and determine whether all loading components that have transmitted the same transaction event log as the current completed transaction have completed execution; wherein, the current transaction is considered a completed transaction after its transaction event log has been processed.

[0023] When all loading components have been executed, clear the preset cache set corresponding to the current completed transaction; obtain at least one ready transaction that has a concurrent conflict with the current completed transaction from the conflict transaction queue, and release the concurrency constraint between each ready transaction and the current completed transaction;

[0024] Determine whether the acquired ready transactions have other concurrency constraints, add ready transactions without concurrency constraints to the execution transaction queue, and transmit the corresponding ready transactions to all subsequent components according to the execution transaction queue;

[0025] Remove the currently completed transaction from the execution transaction queue, retrieve the transaction event log of the next transaction, and continue until a stop retrieval instruction is received.

[0026] Furthermore, the transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; the transformation of the transaction event log through at least one transformation component to obtain target data includes:

[0027] The transaction event log is obtained by transforming the dispatch thread. In the transformation mapping structure, it is queried whether the transaction corresponding to the transaction event log has been assigned to an execution transformation thread.

[0028] If not assigned, an execution transformation thread is assigned to the transaction through the transformation dispatch thread, and the transaction and the corresponding execution transformation thread are written into the transformation mapping structure; wherein, one execution transformation thread processes all transaction event logs of a transaction;

[0029] If already allocated, the transaction event log is passed to the corresponding execution transformation thread via the transformation dispatch thread;

[0030] The allocated transaction event logs are processed by the conversion thread. When the transaction event logs match the user conversion rules, the corresponding conversion operations are performed on the transaction event logs to obtain the target data.

[0031] Further, transmitting the target data to the loading component includes:

[0032] If the number of concurrent transactions exceeds a preset threshold, the target data corresponding to all transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread, or the target data corresponding to the current continuous transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread; wherein, one transaction corresponds to at least one transaction event log;

[0033] If the number of concurrent transactions is less than or equal to a preset threshold, the target data corresponding to the preset number of transaction event logs will be transmitted to the target message queue by executing the conversion thread.

[0034] Specifically, after the target data corresponding to the transaction's commit event has been transmitted, the transaction is deleted from the transformation mapping structure.

[0035] Furthermore, the loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; the step of loading the target data into the database through the loading component includes:

[0036] The loading and distribution thread reads the target data corresponding to the transaction event log from the target message queue, and queries the loading mapping structure to see if the transaction corresponding to the target data has been allocated a database connection object.

[0037] If not allocated, when a free database connection object exists, the free database connection object is allocated to the transaction and recorded in the load mapping structure; the free database connection object is placed in the load ready queue, and the target data is read from the target message queue again until the target message queue is empty; wherein, one execution load thread processes all transaction event logs for one transaction;

[0038] If the target data has been allocated, the target data is passed to the corresponding database connection object. When the database connection object is waiting for the transaction event log, the database connection object is placed in the loading ready queue, and the target data is read from the target message queue again. When the database connection object is not waiting for the transaction event log, the target data is read directly from the target message queue again until the target message queue is empty.

[0039] The loading thread retrieves database connection objects from the loading ready queue and loads the corresponding target data into the database in the order of the transaction event logs within the transaction.

[0040] Furthermore, the step of retrieving database connection objects from the loading-ready queue by executing the loading thread and sequentially loading the corresponding target data into the database according to the order of transaction event logs in the transaction includes:

[0041] If the last target data is not a commit event, after the data is inserted into the database, the current database connection object is placed in the waiting connection queue, and the database connection object is retrieved from the loading ready queue again until the loading ready queue is empty.

[0042] If the last target data is a commit event, and the loading component is the last loading component, then the transaction is added to the loading completion notification queue of the corresponding reading component; if the waiting connection queue is empty, then the current database connection object is made idle; if the waiting connection queue is not empty, then the current database connection object is assigned to the head transaction in the waiting connection queue, and the assigned database connection object is placed in the loading ready queue, and database connection objects are retrieved from the loading ready queue again until the loading ready queue is empty.

[0043] Further, combining at least one reading component and at least one loading component to obtain the desired data synchronization task; or, combining at least one reading component, at least one conversion component, and at least one loading component to obtain the desired data synchronization task includes:

[0044] The connection relationship between at least one read component and at least one load component is determined based on the processing logic of the transaction event log; or, the connection relationship between at least one read component, at least one transformation component, and at least one load component is determined based on the processing logic of the transaction event log.

[0045] Based on the connection relationship, the corresponding components are connected together using directed arrow lines to obtain the data synchronization task;

[0046] Among them, a reading component can be connected to at least one conversion component and / or at least one loading component, a conversion component can be connected to at least one conversion component and / or at least one loading component, and at least one of the loading components can be connected to the destination database.

[0047] Secondly, the present invention also provides a modular data synchronization apparatus for implementing the modular data synchronization method described in the first aspect, the apparatus comprising:

[0048] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the componentized data synchronization method described in the first aspect.

[0049] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the componentized data synchronization method described in the first aspect.

[0050] Unlike existing technologies, the present invention has at least the following beneficial effects:

[0051] This invention combines a reading component, a transformation component, and a loading component to obtain the required data synchronization task. It uses functional components to graphically implement complex data synchronization tasks. The reading component obtains the transaction event logs involved in the data synchronization task, transmits the transaction event logs to the transformation component, transforms the transaction event logs to obtain target data, and transmits the target data to the loading component. The loading component then stores the target data in a database. This invention establishes a transaction synchronization process that includes data transformation, supports rich data transformation operations, and makes the data synchronization system highly scalable. It solves the problems of existing technologies that use attribute files to configure data transformation operations, which are inconvenient to operate and maintain, have low usability, and are very inflexible in task design. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating a formal definition of a DML operation provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating a formal definition of a DDL operation provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating a formal definition of a transaction provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the architecture of an incremental data synchronization system based on transaction logs provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the overall process of a component-based data synchronization method provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the workflow of a reading component provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of a one-to-many data synchronization task provided in an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of a specific process for step 10 of an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of a one-to-one data synchronization task provided in an embodiment of the present invention;

[0062] Figure 10 This is a schematic diagram of a many-to-one data synchronization task provided in an embodiment of the present invention;

[0063] Figure 11 This is a schematic diagram of a many-to-many data synchronization task provided in an embodiment of the present invention;

[0064] Figure 12 This is a schematic diagram of a specific process for step 20 in an embodiment of the present invention;

[0065] Figure 13 This is a schematic diagram of a specific process for step 203 in an embodiment of the present invention;

[0066] Figure 14 This is a schematic diagram of a specific process for step 204 in an embodiment of the present invention;

[0067] Figure 15 This is a schematic diagram of the internal design and workflow of a reading component provided in an embodiment of the present invention;

[0068] Figure 16 This is a schematic diagram of a specific process for step 30 in an embodiment of the present invention;

[0069] Figure 17 This is a schematic diagram of the internal design and workflow of a conversion component provided in an embodiment of the present invention;

[0070] Figure 18 This is another specific flowchart of step 30 in an embodiment of the present invention;

[0071] Figure 19 This is a schematic diagram of a specific process for step 40 in an embodiment of the present invention;

[0072] Figure 20 This is a schematic diagram of the internal design and workflow of a loading component provided in an embodiment of the present invention;

[0073] Figure 21 This is a schematic diagram of the architecture of a modular data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0075] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0076] In this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0077] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium.

[0078] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0079] Example 1:

[0080] To better illustrate the component-based data synchronization method of this invention, the relevant concepts are explained below.

[0081] A transaction is the basic unit of database operations; a transaction consists of one or more events. Specifically, a single DDL event constitutes a transaction, and its execution takes effect immediately. One or more consecutive DML events constitute a transaction, and each execution must ultimately be explicitly committed with a Commit or Rollback event to mark the completion of the transaction, its persistence taking effect, or its rollback, reverting the executed actions. Before persisting the data changes made by a transaction to the data files, the database management system writes the event information of the changed data to a log file (redo / undo log) in an append-only manner.

[0082] Concurrency is a fundamental feature supported by modern database management systems. Concurrency refers to the ability of multiple database users to perform read and write operations simultaneously, that is, multiple transactions are being executed at the same time, and the events of multiple transactions are being written to the log file at the same time.

[0083] Before describing the data synchronization method of this invention, we first define the transaction event log in a formal way: the transaction event log refers to the log events (Events) contained in the transaction. This invention considers two types of transaction event logs: DML events and DDL events.

[0084] like Figure 1 The figure shows a formal definition of a DML event according to an embodiment of the present invention; the content of a DML event also includes a rowID, a unique identifier of its data in the source database; in an optional embodiment, a DML log event can be represented as:<trxId,seqNo,lsn,X,beforeValue,afterValue> .

[0085] Abstractly speaking, DML operation log events only include two types: reading data and writing data. Since this embodiment of the invention only involves data synchronization, we only need to focus on the transaction event logs of data modification operations, i.e., data writing events, which are specifically divided into three event types: adding data, modifying data, and deleting data. All valid DML transactions end with a commit event.

[0086] like Figure 2 The diagram shows the formal definition of a DDL log event. A DDL operation log event is defined as follows: one SQL statement corresponds to one transaction, and one DDL command script constitutes one DDL transaction; that is, one DDL event is one transaction. A DDL log event can be represented as:<trxId,lsn,opScript> .

[0087] like Figure 3 The figure shows the formal definition of a transaction, which consists of an ordered set of events.

[0088] Transaction event logs from different transactions are interleaved in the log file. Transaction event logs from the same transaction appear in the log file in the order they are executed. The database management system does not guarantee that transaction event logs belonging to the same transaction are stored adjacently in the log file.

[0089] Overall, the log file contains the order in which transactions were executed and committed, as well as the order in which transaction event logs within a transaction were executed. Transaction event logs from different transactions are interleaved in the log file. Therefore, when parsing the log file, the transaction event logs obtained in sequence will most likely belong to different transactions.

[0090] Early database management systems (DBMS) did not provide an Application Programming Interface (API) for transaction event logs. Data synchronization systems had to parse log files themselves to extract necessary data change information. For open-source DBMSs such as MySQL and PostgreSQL, log file parsing could be achieved by analyzing source code and design documents. However, for commercial closed-source DBMSs such as Oracle, SQL Server, and DB2, this was difficult and could also pose legal risks, unless the database vendor implemented its own incremental data synchronization system based on its transaction logs. In recent years, database vendors have gradually provided APIs for reading transaction logs, making it possible for third parties to develop incremental data synchronization systems based on transaction logs, such as Oracle's LogMiner interface and PostgreSQL's publish-subscribe functionality based on logical replication.

[0091] like Figure 4 As shown, the architecture of an incremental data synchronization system based on transaction logs generally includes two service components: a source end and a destination end. The source end obtains transaction event logs and sends them to the destination end. The destination end converts the transaction event logs into SQL or other commands and submits them to the destination database server for execution, thereby achieving consistency with the source database data and completing data synchronization. The transaction server (TrxServer), marked with a dashed box, can be an API provided by the source database management system for obtaining transaction logs, or it can be a function implemented by the source service component. The source service component parses the log file to obtain the transaction event logs. The transaction server can also integrate a message server, storing the parsed log files on the message server for multiple source servers or other programs to read.

[0092] like Figure 4 The transaction log incremental data synchronization system shown is used in master-slave database clusters, and its architecture is mostly a master-slave architecture with one master and one slave or one master and multiple slaves. The master-slave architecture is designed to solve the high availability problem of data service systems. If the master system fails, the slave system can immediately take over and provide data services to external applications, requiring complete data consistency between the master and slave databases. Master-slave data synchronization is a typical one-to-one or one-to-many data synchronization application, where the master and slave databases have completely identical data.

[0093] Since many commercial closed-source databases often require data transformation during data synchronization, the data in the source database (i.e., the data generation database) cannot be synchronized to the destination database unchanged. Therefore, it is necessary to transform the data in the source database in a certain way before synchronizing it to the destination database.

[0094] Existing technologies generally do not support data transformation operations, or can only achieve certain forms of data transformation through relatively complex attribute file configurations. This results in poor usability and scalability, and a lack of rich and easy-to-use data transformation operations. Furthermore, the deployment of servers used to implement data transformation is complex. For example, when deploying a one-to-many data synchronization task, multiple independent destination servers must be configured, greatly increasing the complexity and difficulty of monitoring the data synchronization system. This hinders fault monitoring, recovery, and maintenance of the data synchronization system, resulting in poor practicality.

[0095] For complex data synchronization tasks, existing technologies lack independent and flexible design support for task design; designing data synchronization tasks through configuration files or abstract data transformation function interfaces lacks a visual and graphical design approach, making it inflexible and unintuitive.

[0096] To solve the aforementioned problems, such as Figure 5 As shown, this embodiment of the invention provides a component-based data synchronization method, including:

[0097] Step 10: Combine at least one read component and at least one load component to obtain the required data synchronization task; or, combine at least one read component, at least one transformation component and at least one load component to obtain the required data synchronization task.

[0098] This invention employs functional components to design data synchronization tasks. Each functional component provides a single function and includes three types: reading components, transformation components, and loading components. Specifically, the reading component acquires transaction event logs, the transformation component transforms the transaction event logs, and the loading component applies the transaction event logs to the destination database for execution.

[0099] This invention obtains the required data synchronization task by selecting at least one reading component and at least one loading component, and combining them with a conversion component selected from the two according to usage requirements. It also supports users designing data synchronization tasks by dragging and dropping functional graphical components. Logically, transaction event logs flow and are processed between various functional components, enabling support for complex data synchronization tasks. Using the component-based data synchronization method of this invention, data synchronization can be achieved without complex attribute file configuration operations for the reading, conversion, and loading components.

[0100] Based on the requirements of the data synchronization task, no conversion component is needed between the read component and the load component in some or all branches of the data synchronization task process. In the scenario of primary-standby architecture (one primary and one standby or one primary and multiple standby), the standby data is completely consistent with the primary data, and no data conversion is required. Therefore, there is no need to use a conversion component, and the read component and load component of the present invention can be directly combined to achieve data synchronization.

[0101] Step 20: Obtain the transaction event logs involved in the data synchronization task through the reading component, and transmit the transaction event logs to all subsequent components of the reading component; wherein, the subsequent components of the reading component are transformation components or loading components.

[0102] The reading component in this embodiment of the invention collects complete transaction event logs, performs concurrent conflict detection, distributes the complete transaction event logs, and cleans up the transaction event logs (i.e., removes target data from the cache). Specifically, for small transactions involved in the data synchronization task, the reading component distributes all transaction event logs for that transaction each time, and in optional embodiments, transmits them in the form of transaction event log groups; for transactions containing large fields or long transactions, the reading component distributes a portion of the continuously executed transaction event logs for that transaction each time.

[0103] Retrieve transaction event logs from the transaction server and cache them, such as... Figure 6 As shown, when distributing the complete transaction event log to subsequent components, a concurrency conflict check must be performed with the previously distributed and currently executing transactions. If no conflict exists, the conflict-free transaction event log is submitted to the subsequent component; if a conflict exists, the transaction to which the conflicting transaction event log belongs is mounted. The subsequent component of the reading component (i.e., the functional component connected to the reading component) can be a transformation component or a loading component. Furthermore, each time the synchronous execution of a transaction is completed, the reading component is responsible for clearing all transaction event log data for that transaction from the cache.

[0104] Step 30: When the subsequent component of the reading component is a transformation component, the transaction event log is transformed by at least one transformation component to obtain the target data; the target data is transferred to the loading component; and the target data is stored in the database by the loading component.

[0105] In a data synchronization task, multiple transformation components can be cascaded. Each transformation component operates internally with identical logic; the only differences lie in the data source and data flow. When only one transformation component exists between the read and load components, its data source is the read component, and it transmits the transformed target data to the load component. When more than one transformation component exists between the read and load components, for a transformation component connected to the read component, its data source is the read component, and it transmits the transformed target data to the next transformation component connected to it; for a transformation component connected to a conversion component, its data source is the conversion component, and it transmits the further transformed target data to the next transformation component connected to it or the load component connected to it.

[0106] In this embodiment of the invention, each transformation component transforms and distributes the transaction event log. It obtains the complete transaction event log from the upstream component (i.e., the reading component or the previous transformation component with which it is connected) and performs data transformation operations on the transaction event log data according to user-defined rules and methods. After the transformation is complete, the transformed target data is submitted to the subsequent component. The subsequent component can be either a transformation component or a loading component, depending on the specific task design of the data synchronization task.

[0107] Step 40: When the subsequent component of the reading component is the loading component, the target data is loaded into the database through the loading component.

[0108] The loading component in this embodiment of the invention utilizes a database connection pool to achieve concurrent loading, ultimately obtaining a successful transaction loading feedback. The loading component atomically submits the complete transaction event log to the destination database, completing data synchronization. For example, for a relational database, the loading component converts DML log events into SQL statements and submits them to the destination database for execution; DDL log events are directly submitted to the destination database for execution, thus completing the data synchronization of one transaction. When multiple transactions are continuously sent from the upstream component, the loading component supports concurrent loading of transactions. After the transaction loading is complete, feedback needs to be sent to the corresponding reading component, which then clears the cached transaction event log data accordingly.

[0109] This invention combines a reading component, a transformation component, and a loading component to obtain the required data synchronization task. It uses functional components to graphically implement complex data synchronization tasks. The reading component acquires the transaction event logs involved in the data synchronization task and transmits them to the transformation component. The transformation component transforms the transaction event logs to obtain target data, which is then transmitted to the loading component. The loading component stores the target data in a database. Upon successful storage, the reading component is notified. This establishes a transaction synchronization process that includes data transformation, supports rich data transformation operations, and enhances the scalability of the data synchronization system. It solves the problems of existing technologies that use attribute files for data transformation configuration, which are inconvenient to operate and maintain, have low usability, and are inflexible in task design.

[0110] To address the problems associated with configuring data transformation operations using property files, this invention provides a component-based incremental data synchronization method. Specifically, as follows: Figure 8 As shown, the functional components include a reading component, a conversion component, and a loading component. Step 10 specifically includes:

[0111] Step 101: Determine the connection relationship between at least one read component and at least one load component according to the processing logic of the transaction event log; or, determine the connection relationship between at least one read component, at least one transformation component and at least one load component according to the processing logic of the transaction event log.

[0112] This invention designs data synchronization tasks by combining different functional components. The data synchronization system processes data according to the user-designed data synchronization tasks; each functional component runs on an independent thread. Data is transferred between functional components using a synchronization queue. The data transferred between functional components consists of all transaction event logs or partially consecutively executed transaction event logs belonging to the same transaction. Data flowing between two functional components belongs to the same transaction. For small transactions, a single data flow between functional components includes all transaction event logs of that transaction; for transactions containing large fields or long transactions, it generally only includes partially consecutively executed transaction event logs of that transaction.

[0113] Step 102: Connect the corresponding components together with directed arrows according to the connection relationship to obtain the data synchronization task.

[0114] In this configuration, a read component can be connected to at least one transformation component and / or at least one load component, a transformation component can be connected to at least one transformation component and / or at least one load component, and at least one load component can be connected to the destination database. In an optional embodiment, the source database is connected to at least one read component, the transformation component is generally connected to only one read component or transformation component, and the load component is generally directly connected to the destination database (such as a central database or a relational database).

[0115] Based on the connection rules of the above functional components, a wide variety of data synchronization tasks can be designed: a one-to-one relationship can be achieved between the read component and the load component (e.g., ...). Figure 9 As shown), one-to-many (e.g.) Figure 7 As shown), many-to-one (e.g.) Figure 10 As shown), many-to-many (e.g.) Figure 11 Data synchronization tasks, such as those involving reading and loading components, may have no, one, or multiple conversion components between each pair of reading and loading components. The data processing output of each conversion component can be shared with all directly subsequent functional components (including subsequent conversion and loading components). In optional embodiments, hybrid data synchronization tasks can also be designed, where multiple transaction synchronization types, such as one-to-one, one-to-many, many-to-one, and many-to-many, coexist within a single data synchronization task. Figure 7 and Figures 9-11 The four data synchronization tasks shown (i.e., transaction synchronization processes) can be merged into a single data synchronization task (i.e., transaction synchronization process).

[0116] like Figure 9 The image shows a typical one-to-one data synchronization task.

[0117] Among them, SrcDB01, SrcDB02, and SrcDB03 are reading components; Trans01, Trans02, Trans03, Trans04, Trans05, and Trans06 are transformation components; SinkDB01, SinkDB02, and SinkDB03 are loading components; DBMS stands for Database Management System, and RDBMS stands for Relational Database Management System.

[0118] like Figure 7The diagram illustrates a typical one-to-many data synchronization task, consisting of one source database and three destination databases. The source database is a relational database, while each destination database comprises a relational database, an Elasticsearch search engine, and a Kafka message server. The data synchronization system synchronizes changed data from the source database to the destination databases. The relational database within the destination databases serves as a backup for the source database, forming a master-slave architecture. The Elasticsearch search engine handles search services from external applications, and the Kafka server provides message subscription services. Considering requirements for data confidentiality and data format consistency, the data synchronized to the Elasticsearch search engine and Kafka cannot be directly used as the original data; it requires a transformation. This is achieved by adding transformation components Trans01 and Trans02. Trans01's upstream component is the read component SrcDB01, and its downstream component is the load component SinkDB02. Similarly, Trans02's upstream component is the read component SrcDB01, and its downstream component is the load component SinkDB03.

[0119] like Figure 10The diagram illustrates one form of data integration, a typical many-to-one data synchronization task. Data integration migrates data from different databases to a central database (i.e., the central database), which provides data services for data analysis. Data from the source databases generally cannot be directly used for data analysis and requires certain transformations. In this embodiment of the invention, one or more transformation operations occur between the source database and the central database, executed by transformation components Trans01, Trans02, Trans03, Trans04, Trans05, and Trans06. Among them, the upstream component of conversion component Trans01 is the read component SrcDB01, and the downstream component is conversion component Trans02; the upstream component of conversion component Trans02 is conversion component Trans01, and the downstream component is conversion component Trans03; the upstream component of conversion component Trans03 is conversion component Trans02, and the downstream component is loading component SinkDB01; conversion components Trans01, Trans02, and Trans03 are cascaded; the upstream component of conversion component Trans04 is the read component SrcDB02, and the downstream component is conversion component Trans05; the upstream component of conversion component Trans05 is conversion component Trans04, and the downstream component is loading component SinkDB01; conversion components Trans04 and Trans05 are cascaded; the upstream component of conversion component Trans06 is the read component SrcDB03, and the downstream component is loading component SinkDB01.

[0120] like Figure 11The diagram illustrates another form of data integration: a many-to-many data synchronization task. Change data from three production databases (three source databases) is transformed in different ways and then loaded into three central databases via loading components, achieving data integration. Specifically, the upstream component of transformation component Trans01 is the read component SrcDB01, and the downstream component is transformation component Trans02; the upstream component of transformation component Trans02 is transformation component Trans01, and the downstream component is transformation component Trans03; the upstream component of transformation component Trans03 is transformation component Trans02, and the downstream components are loading components SinkDB01 and SinkDB02; loading component SinkDB01 loads the target data obtained from transformation component Trans03 into central database 1, and loading component SinkDB02 loads the target data obtained from transformation component Trans03 into central database 2; transformation components Trans01, Trans02, and Trans03 are cascaded. The upstream component of transformation component Trans04 is the read component SrcDB02, and the downstream component is transformation component Trans05. The upstream component of transformation component Trans05 is transformation component Trans04, and the downstream component is loading component SinkDB02. Loading component SinkDB02 loads the target data obtained from transformation component Trans05 into central database 2. Transformation components Trans04 and Trans05 are cascaded. The upstream component of transformation component Trans06 is the read component SrcDB03, and the downstream components are loading components SinkDB02 and SinkDB03. Loading component SinkDB02 loads the target data obtained from transformation component Trans06 into central database 2. Loading component SinkDB03 loads the target data obtained from transformation component Trans06 into central database 3.

[0121] The internal operating logic of the data synchronization task is explained below:

[0122] During data synchronization, each functional component starts an independent execution thread. Transaction event log data can be passed between functional components and their directly connected counterparts using message queues. A pipelined data processing process for the read component consists of a transaction acquisition thread, a conflict detection thread, a distribution thread, and a cleanup thread. Transaction event logs within the same transaction are processed by the same transaction transformation thread, with multiple transformation threads concurrently performing data processing in the transformation component. Transactions are bound to database connection objects, and transaction event logs within the same transaction are submitted to the destination database through the same database connection object. Multiple execution loading threads concurrently perform data processing in the loading component. Each read component, each transformation component, and each loading component runs using an independent execution thread. Based on the data processing processes of the read component, the transformation component, and the loading component, a data synchronization task is executed to complete the data synchronization.

[0123] To further illustrate the internal process of reading the component, such as Figure 12 As shown, step 20 specifically includes:

[0124] Step 201: Obtain the transaction event logs belonging to the same transaction, and store the transaction event logs belonging to the same transaction in the corresponding preset cache set. The transaction event logs belonging to the same transaction are either all transaction event logs of the transaction or a portion of consecutive transaction event logs.

[0125] The preset cache set is used to cache collected transaction event logs, and its specific implementation can be selected by those skilled in the art based on the specific use case. In an optional embodiment, the transaction event logs are cached in the form of a list, and a hash table can be used as the preset cache list.

[0126] In this embodiment of the invention, for long transactions and transactions containing large field data, the transaction event logs collected by the reading component need to be cached in external storage and cannot all be in memory. Furthermore, the reading component sends a portion of the continuous transaction event logs of the transaction to subsequent components. For non-long transactions and transactions that do not contain large field data, the reading component sends all transaction event logs belonging to the transaction. In an optional embodiment, all transaction event logs of the same transaction are sent to subsequent components in the form of transaction event log groups.

[0127] Step 202: When the transaction event log is a commit event, add the transaction event log corresponding to the commit event from the preset cache set to the transmission ready queue to obtain a ready transaction.

[0128] The reading component continuously retrieves transaction event logs from the outside world. When a transaction commit event is read, it indicates that the complete transaction event log for that transaction has been read, meaning that the source database has successfully executed the transaction. The data synchronization system should then persist the transaction event log to the destination database.

[0129] In an optional embodiment, the data synchronization system can issue complete transaction event logs in a serial manner, that is, a transaction is distributed to subsequent downstream components and finally issued after all loaded components have completed execution.

[0130] Step 203: Based on whether there are concurrent conflicts among the ready transactions in the ready queue, selectively transmit the corresponding ready transactions to all subsequent components; or attach the corresponding ready transactions to the conflict transaction queue until the transactions that have concurrent conflicts with the ready transactions are completed, then remove the ready transactions from the conflict transaction queue and transmit them to all subsequent components.

[0131] Because database management systems (DBMS) often support concurrent execution, allowing multiple transactions to run simultaneously, and because DBMS guarantees that all successfully executed transactions are non-conflicting, it appears as if transactions are executed sequentially one after another. This is reflected in the transaction log file, where the commit event of the first completed transaction is listed first. This is the basis for the feasibility of incremental data synchronization systems based on log events executing multiple transactions concurrently. Therefore, a transaction concurrency (or parallelism) conflict detection module needs to be added to the reading component to detect whether there are any concurrent execution conflicts between the currently collected complete transaction event log and the transactions that are being processed.

[0132] A transaction consists of multiple ordered events that modify data objects, such as adding, deleting, and updating data. If two transactions contain events that modify the same data object, then these two transactions will have a concurrency conflict. The subsequent transaction can only be executed after the preceding transaction has completed. After the current transaction event log is collected, it needs to be checked for concurrency conflicts with previously executing transactions and transactions that were suspended due to concurrency conflicts. If no concurrency conflicts are found, the transaction can be published from the read component to its direct subsequent component (such as the load component). During the concurrency conflict detection process, the transaction IDs of all transactions that conflict with the current transaction are additionally retained for the transaction to be checked.

[0133] like Figure 6 As shown, in an optional embodiment, when concurrency conflicts exist, the transaction is attached to the transaction conflict waiting list; when there are no conflicts, the transaction is sent from the read component to all subsequent components.

[0134] Step 204: After the transaction event log of the current transaction is processed, the transaction event log of the next transaction is obtained, until a stop acquisition instruction is received.

[0135] After all loading components have finished executing each complete transaction, the reading component will notify the reading component that the transaction has been completed. Upon receiving the notification that the transaction has been completed, the reading component will perform two operations: First, it will clear the information of all transaction event logs of the transaction in the preset cache set and release the cache; second, it will traverse sequentially from the head of the conflict transaction queue. If the ready transactions waiting in the conflict transaction queue only conflict with the currently completed transaction, then the ready transactions in the conflict transaction queue can be issued to subsequent components.

[0136] Operational details not mentioned in the embodiments of this invention shall be selected by those skilled in the art based on the specific application scenario and without creative effort, and shall not be elaborated here.

[0137] To further explain concurrent conflict detection, such as Figure 13 As shown, in step 203, the step of selectively transmitting the corresponding ready transaction to all subsequent components based on whether there are concurrent conflicts in the ready transaction queue; or attaching the corresponding ready transaction to the conflict transaction queue includes:

[0138] Step 2031: Sequentially determine whether there are any transaction event logs in the execution transaction queue that conflict with the current ready transaction; when there is a conflict, record all transaction event logs that conflict with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue.

[0139] Step 2032: Sequentially determine whether there are any transaction event logs in the conflict transaction queue that have concurrent conflicts with the current ready transaction; when there are concurrent conflicts, record all transaction event logs that have concurrent conflicts with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue.

[0140] In an optional embodiment, each ready transaction in the conflict transaction queue that has concurrent conflicts has a corresponding list of related transactions, which records the ready transactions that have concurrent conflicts with that ready transaction.

[0141] Each time a concurrent conflict is detected, the transaction IDs corresponding to all concurrent conflict transaction event logs are recorded in the transaction list of the current ready transaction, and the transaction ID of the current ready transaction is attached to the conflict transaction queue.

[0142] Step 2033: When there is no concurrent conflict in the current ready transaction, add the current ready transaction to the execution transaction queue, and transmit the ready transaction to all subsequent components according to the execution transaction queue.

[0143] To free up cache space occupied by transaction event logs that have already completed data synchronization, such as Figure 14 As shown, step 204 specifically includes:

[0144] Step 2041: Obtain the completed transactions in the loading completion notification queue, and determine whether all loading components that transmit the same transaction event log as the current completed transaction have been executed; wherein, the current transaction is a completed transaction after the transaction event log of the current transaction has been processed.

[0145] Among them, the completed transaction refers to the transaction that has been executed and inserted into the database by the loading component; after the loading component inserts the transaction into the database, it returns a loading completion notification queue to the corresponding reading component to notify the reading component to clear the cache occupied by the executed and inserted transactions (the preset cache set corresponding to the completed transaction) and release the cache space.

[0146] In this embodiment of the invention, the cache occupied by completed transactions is cleared through a transaction completion counter. Each transaction ID in the transaction completion counter corresponds to a counter value that records the number of transaction event logs that the transaction needs to be entered into the database.

[0147] Each time a transaction ID of one or more completed transactions is read from the load completion notification queue, the counter value corresponding to that transaction ID is obtained from the transaction completion counter, and the counter value is reduced by one unit count (i.e., 1). The transaction IDs in the load completion notification queue are read in a loop. Each time, the counter value corresponding to the transaction ID is reduced in the transaction completion counter according to the number of transaction event logs entered by the transaction ID, until the counter value is 0.

[0148] In an optional embodiment, when the counter value is 0, that is, after the commit event has been entered into the database and the counter value has been reduced through the loading completion notification queue, the completed transaction is placed at the head of the ready queue, and the counter item corresponding to the completed transaction is cleared from the transaction completion counter. This allows ready transactions in the conflict transaction queue that have concurrent conflicts with the current completed transaction to be immediately transmitted to the transformation message queue (when the subsequent component is a transformation component) or the target message queue (when the subsequent component is a loading component). In addition, all transaction event log data related to the completed transaction are cleared from the preset cache list corresponding to the transaction ID.

[0149] Step 2042: When all loading components have been executed, clear the preset cache set corresponding to the current completed transaction; obtain at least one ready transaction that has a concurrent conflict with the current completed transaction from the conflict transaction queue, and release the concurrency constraint between each ready transaction and the current completed transaction.

[0150] In an optional embodiment, each ready transaction in the conflict transaction queue maintains a conflict list. Each ready transaction's conflict list records the transaction IDs of transactions that have concurrent conflicts with it, i.e., it records its own concurrency constraints. The conflict transaction queue is traversed to identify at least one ready transaction in the queue that has a concurrent conflict with the currently completed transaction. The transaction ID of the currently completed transaction is removed from the conflict list of each ready transaction with a concurrent conflict, thereby releasing the concurrency constraint between the ready transaction with the concurrent conflict and the currently completed transaction.

[0151] Step 2043: Determine whether the acquired ready transactions have other concurrency constraints, add ready transactions without concurrency constraints to the execution transaction queue, and transmit the corresponding ready transactions to all subsequent components according to the execution transaction queue.

[0152] In an optional embodiment, a ready transaction whose conflict list is empty after the transaction is removed (there are no other concurrency constraints) is identified, the ready transaction with an empty conflict list is added to the execution transaction queue and sent to the transformation message queue or the target message queue, and the ready transaction with an empty conflict list is added to the transaction completion counter.

[0153] Step 2044: Remove the currently completed transaction from the execution transaction queue, obtain the transaction event log of the next transaction, until a stop acquisition instruction is received.

[0154] It should be noted that the currently completed transaction may not have any concurrent conflicts with any ready transaction in the conflict transaction queue. In this case, the currently completed transaction is directly removed from the execution transaction queue, and the transaction event log of the next transaction is obtained iteratively until a stop acquisition instruction is received.

[0155] This invention provides an optional solution for reading the internal design of a component, such as... Figure 15 As shown, the details are as follows:

[0156] A hash table is used to cache and collect transaction event logs. The transaction ID (Trxid) is used as the key of the hash table, and the elements of the hash table are linked lists of transaction event logs belonging to the same transaction.

[0157] The main components include: data structures such as Hash cache, transmission ready queue, conflict transaction queue, execution transaction queue, and transaction completion counter; and three levels of processing threads such as transaction acquisition thread, conflict detection thread, and cleanup thread.

[0158] The main components of the reading component are described below:

[0159] Hash cache: Stores all collected transaction event logs, with transaction event logs belonging to the same transaction organized together;

[0160] Transmission Ready Queue: Stores the transaction IDs of transactions whose transaction event logs have been fully collected. When a transaction commit event is collected, it is considered that all events of that transaction have been collected, and the transaction ID is appended to the "Transmission Ready Queue".

[0161] Conflicting Transaction Queue: When a transaction conflicts with a previous transaction, the transaction ID is appended to the "Conflicting Transaction Queue" to wait for the conflicting transaction to complete.

[0162] Execution Transaction Queue: Once a transaction has been fully collected and there are no concurrency conflicts with previous transactions, it can be sent to subsequent components. The transaction ID is stored in the "Execution Transaction Queue".

[0163] Transaction Completion Counter: This counter represents the number of loading components corresponding to the read component. Transactions collected by the read component are ultimately committed to the destination database by the loading components. Only after all loading components have committed and executed can the transaction event log data in the hash cache be cleared. The "Transaction Completion Counter" is a mapping structure where the key is the transaction ID, and the initial value (counter value) is the number of loading components corresponding to the read component. All initial values ​​are the same. After the destination component completes a transaction, it returns the transaction ID to the read component, and the counter value is decremented by 1. If the counter value is 0, it indicates that the transaction has been fully executed.

[0164] Transaction acquisition thread:

[0165] The three main working steps of the transaction acquisition thread are as follows: Figure 15 Steps 1, 2, and 3 shown are as follows:

[0166] All steps are shown in parentheses, for example, step 1 is shown as “(1)”, and the same applies below.

[0167] Step 1: Obtain transaction event logs from external sources;

[0168] Step 2: Store the transaction event log in the linked list corresponding to the Hash cache based on the transaction ID;

[0169] Step 3: Collect the complete transaction ID from the transaction event log and add it to the transport ready queue;

[0170] The transaction acquisition thread retrieves transaction event logs from the outside world via (step 1). Each retrieved transaction event log is cached in a hash table via (step 2). If the currently processed transaction event log is a committed transaction, the corresponding transaction ID is added to the transmission ready queue. After the currently collected transaction event log is processed, (step 1) is executed again for the next round of acquisition, until a stop command is received.

[0171] Conflict detection thread:

[0172] The main working steps of the conflict detection thread are as follows: Figure 15 The steps shown are 4, 5, 6, 7, 8, 9, 10, 16, and 17, as detailed below:

[0173] Step 4: Retrieve complete ready or completed transactions from the log ready queue. If no transaction is retrieved, block and wait. If a ready transaction is retrieved, proceed from (Step 5) to (Step 10). If a completed transaction is retrieved, proceed from (Step 16) to (Step 17). After processing, retrieve transactions from the ready queue again and repeat the process until a stop command is received.

[0174] Step 5: Traverse all events in the execution transaction queue, perform transaction concurrency conflict detection with the current ready transaction, and if a conflict exists, record all conflicting transaction IDs in the ready transaction;

[0175] Step 6: Traverse all events in the conflict transaction queue and perform transaction concurrency conflict detection with the current ready transaction. If a conflict exists, record all conflicting transaction IDs in the conflict list of the ready transaction.

[0176] Step 7: If there are no concurrent conflicts in the ready transaction, the ready transaction is sent to the transformation message queue or target message queue of the subsequent component.

[0177] Step 8: If there are no concurrent conflicts in the ready transactions, append the transaction ID of the ready transactions to the execution transaction queue;

[0178] Step 9: If there are no concurrent conflicts in the ready transactions, add the transaction ID of the ready transactions to the transaction completion counter;

[0179] Step 10: If there are no concurrent conflicts in the ready transactions, append the transaction ID of the ready transactions to the conflicting transaction queue and wait for the conflicting transactions to complete.

[0180] Step 16: Traverse the conflict transaction queue. If there is a concurrent conflict between a transaction in the conflict transaction queue and a completed transaction, remove the transaction ID of the completed transaction from the conflict list of that transaction. If the conflict list is empty after removal, the transaction can be issued, i.e., execute (Step 7) and (Step 9).

[0181] Step 17: Remove the transaction IDs of completed transactions from the transaction execution queue.

[0182] Cleanup thread: After a transaction is completed by subsequent components, the last loading component sends a notification of completion to the reading component. This notification is stored in a loading completion notification queue. Combined with a transaction completion counter, it can be determined whether the transaction has been fully loaded, thus clearing its data and releasing transactions waiting in the conflicting transaction queue. The main steps of the cleanup thread are as follows: Figure 15 Steps 11, 12, 13, 14, and 15 are detailed below:

[0183] Step 11: Retrieve the transaction ID of the completed transaction from the loading completion notification queue. If no ID is found, block and wait. After retrieving the ID, execute steps (12) to (15). After execution, reread the loading completion notification queue and repeat the process.

[0184] Step 12: Obtain the corresponding counter value from the transaction completion counter using the transaction ID of the completed transaction, and decrement it by 1. If the result is 0, execute (Step 13), (Step 14) and (Step 15); otherwise, return to (Step 11).

[0185] Step 13: Place the transaction ID of the completed transaction at the head of the ready queue;

[0186] Step 14: Clear the counter entry corresponding to the completed transaction from the transaction completion counter;

[0187] Step 15: Clear all transaction event log data for the completed transaction from the Hash cache.

[0188] To further illustrate the internal process of the conversion component, such as Figure 16 As shown, the transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; in step 30, the transformation of the transaction event log through at least one transformation component to obtain target data includes:

[0189] Step 301a: Obtain the transaction event log through the transformation dispatch thread, and in the transformation mapping structure, query whether the transaction corresponding to the transaction event log has been assigned to an execution transformation thread.

[0190] like Figure 17As shown, this embodiment of the invention uses a two-level thread structure to run the conversion component: a conversion dispatch thread (e.g., ... Figure 17 dThread in the middle) and the thread that performs the conversion (such as dThread) Figure 17 In the `eThread`, only one transformation dispatch thread is needed, while multiple transformation execution threads can be set. A transformation mapping structure (txid-to-eThread-map) is maintained, which maps transaction IDs (Trxid) to the transformation execution threads. This two-level thread structure enables concurrent processing of multiple transactions. The dashed box in the middle shows the internal design of the transformation component.

[0191] The transformation component transforms the data in the transaction event log, with each transaction event log corresponding to a change to a data object. For relational databases, a transaction event log corresponds to a change to one row of data, typically including the schema name, table name, array of column names, array of new column values, and array of old column values. The transformation component transforms the above data content in the transaction event log, such as replacing the value of a column in the array of new column values, renaming a column in the array of column names, and modifying the table name and schema name.

[0192] The conversion component is internally designed with multiple worker threads. One conversion thread processes the transaction event logs of the same transaction, ensuring the order of the transaction event logs within the same transaction. After processing, the target data after the corresponding transaction event log conversion is sent out immediately, or all the target data of the transaction is sent out after all the transaction event logs of a transaction have been processed, reducing the number of times data is transferred between upstream and downstream components.

[0193] Step 302a: If not allocated, an execution transformation thread is allocated to the transaction through the transformation dispatch thread, and the transaction and the corresponding execution transformation thread are written into the transformation mapping structure.

[0194] Step 303a: If allocated, the transaction event log is passed to the corresponding execution transformation thread via the transformation dispatch thread.

[0195] In this embodiment of the invention, after the transformation distribution thread retrieves the transaction event log from the transformation message queue (step 1), it checks in the transformation mapping structure whether the transaction has been assigned to an execution transformation thread (i.e., step 2). If it has been assigned, the transaction event log is passed to the execution transformation thread (i.e., step 4); if it has not been assigned, an execution transformation thread is selected according to a preset rule for processing the current transaction, and the transaction ID and the execution transformation thread are written into the transformation mapping structure (i.e., step 3). Steps 2 and 3 ensure that the transaction event logs of the same transaction are executed by the same execution transformation thread, thereby guaranteeing the orderly processing of the transaction event logs.

[0196] Step 304a: Process the allocated transaction event logs by executing the transformation thread. When the transaction event logs match the user transformation rules, perform the corresponding transformation operation on the transaction event logs to obtain the target data.

[0197] The transformation thread processes the transaction event log or a portion of the transaction event log. When the transaction event log matches the user-defined transformation rules, the corresponding transformation operation is performed on the data in the transaction event log; otherwise, no transformation is performed. After the transaction event log is processed, it is sent to subsequent components (i.e., step 5). After processing all transaction event logs for a transaction, i.e., after transforming the commit event, the transformation thread (eThread) removes the mapping item for that transaction from the transformation mapping structure (i.e., step 6).

[0198] To reduce the performance overhead of the transformation dispatch thread querying the transformation map structure, specifically, as follows: Figure 18 As shown, in step 30, sending the target data to the loading component includes:

[0199] Step 301b: If the number of concurrent transactions is greater than a preset threshold, the target data corresponding to all transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread, or the target data corresponding to the current continuous transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread; wherein, one transaction corresponds to at least one transaction event log.

[0200] The preset threshold is selected by those skilled in the art based on the specific use case, and is not limited here.

[0201] Step 302b: If the number of concurrent transactions is less than or equal to a preset threshold, the target data corresponding to the preset number of transaction event logs is transmitted to the target message queue by executing the conversion thread.

[0202] Specifically, after the target data corresponding to the transaction's commit event has been transmitted, the transaction is deleted from the transformation mapping structure.

[0203] In the message transmission process of this invention embodiment, the functional components transmit transaction event log data through a synchronization queue. The reading component only sends out the transaction after collecting the complete transaction event log and passing the concurrency conflict detection. In principle, all transaction event logs of a transaction are transmitted at one time. Essentially, what is transmitted is the memory reference or memory address of each item in the preset cache set. However, for long transactions or large field transactions, the transaction event logs may need to be cached in external storage. During transmission, only a portion of the transaction event logs may be in the cached memory.

[0204] When the transformation component passes transaction event logs to subsequent components, it can process one transaction log at a time and pass it on. Subsequent components will receive the processed transaction event logs from the upstream component immediately and process them accordingly, but this increases the number of message passes between components. Alternatively, the transformation component can process all transaction event logs belonging to the same transaction and then pass them to subsequent components together, requiring only one message pass per transaction. A compromise is batch passing, which passes consecutive portions of transaction event logs at once. The transformation component has three options for passing transaction event logs to subsequent components. The aforementioned internal design and implementation schemes support these three options. Based on reducing the number of passes and minimizing the frequency of the transformation thread stopping and waiting for transaction event logs, the priority is to pass complete transaction event logs, followed by batch passing of partially consecutive log transactions, and only lastly, passing one transaction event log at a time.

[0205] If the number of concurrent transactions exceeds a preset threshold, the transformation component sends all transaction event logs or only a portion of the transaction event logs to subsequent components. If the number of concurrent transactions is below the preset threshold, the transformation component sends a single processed transaction event log to subsequent components. Excessive concurrent transactions increase the performance overhead of querying the transformation mapping structure by the transformation dispatch thread. By packaging all or a portion of the transaction event logs into a single package and sending them in batches, both the number of queries and the number of message passes between functional components can be reduced. However, this increases the processing wait time for individual transaction event logs. Here, a preset threshold is set to balance the overhead between the two and maximize processing performance.

[0206] To further illustrate the internal process of loading components, such as Figure 19 As shown, specifically, the loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; in steps 30 and 40, the step of loading the target data into the database through the loading component includes:

[0207] Step 401: Read the target data corresponding to the transaction event log from the target message queue, and query in the load mapping structure whether the transaction corresponding to the target data has been allocated a database connection object.

[0208] The loading component maintains a database connection pool. A transaction allocates a database connection object from the pool, and all transaction event logs for a transaction are submitted to the destination database for execution via this database connection object. The loading component runs using a two-tiered thread structure: a loading and dispatch thread (only one is needed) and multiple execution loading threads.

[0209] Step 402: If no connection is allocated, when a free database connection object exists, allocate the free database connection object to the transaction and record it in the load mapping structure; place the free database connection object in the load ready queue, and read the target data from the target message queue again until the target message queue is empty. A log of all transaction events for one transaction is maintained by one execution load thread.

[0210] In an optional embodiment, the database connection pool maintains three queues: a loaded ready queue, an idle queue, and a waiting connection queue.

[0211] Specifically, after the load and dispatch thread retrieves the target from the target message queue, if the transaction ID of the transaction to which the transaction event log belongs has not been allocated a database connection object, and if there are idle connection objects in the database connection pool (i.e., the idle queue is not empty), then a database connection object is retrieved from the database connection pool, allocated to the transaction, and placed into the load ready queue. If there are no idle database connection objects (i.e., the idle queue is empty), then the transaction is placed in the waiting connection queue.

[0212] Step 403: If the target data has been allocated, the target data is passed to the corresponding database connection object; when the database connection object is waiting for the transaction event log, the database connection object is placed in the loading ready queue, and the target data is read from the target message queue again; when the database connection object is not waiting for the transaction event log, the target data is read directly from the target message queue again until the target message queue is empty.

[0213] Step 404: Retrieve database connection objects from the loading queue by executing the loading thread, and sequentially load the corresponding target data into the database according to the order of the transaction event log in the transaction.

[0214] The loading component directly or indirectly submits the transaction event log to the destination database for execution. The execution loading thread retrieves a database connection object from the loading ready queue and executes the transaction event log within that database connection object. Each time a database connection object is retrieved from the ready queue, it submits the insertion command to the destination database one by one according to the order of the transaction event log in the transaction, and inserts the transaction into the database according to the corresponding insertion command. If the waiting connection queue is empty, the database connection object is returned to the database connection pool. The execution loading thread continues to retrieve database connection objects from the ready queue, and executes the transaction assigned to the database connection object after retrieval; otherwise, it waits.

[0215] In an optional embodiment, for a relational database, a transaction event log can generate an SQL statement. The SQL statement is submitted to the destination database for execution. All transaction event logs of the same transaction generate SQL statements in sequence and submit them to the destination database for execution. Then, a commit command is submitted. Only after successful execution is the synchronization of a transaction considered complete.

[0216] If the last target data is not a commit event, after the data is inserted into the database, the current database connection object is placed in the waiting connection queue, and a database connection object is retrieved again from the loading ready queue until the loading ready queue is empty. The waiting connection queue is ordered, and this order corresponds to the storage order.

[0217] If the last target data is a commit event, and the loading component is the last loading component, then the transaction is added to the loading completion notification queue of the corresponding reading component; if the waiting connection queue is empty, then the current database connection object is made idle; if the waiting connection queue is not empty, then the current database connection object is assigned to the head transaction in the waiting connection queue, and the assigned database connection object is placed in the loading ready queue, and database connection objects are retrieved from the loading ready queue again until the loading ready queue is empty.

[0218] After the transaction is completed (commit event is entered into the database), the database connection object and the transaction allocation relationship are released. If the waiting connection queue is not empty, the head transaction is taken out from the waiting connection queue, the head transaction is bound to the database connection object and committed to the ready queue.

[0219] After each loading is completed, the loading component also needs to notify the corresponding reading component of the transaction ID of the completed transaction. The reading component then sends out ready transactions that have concurrent conflicts with the completed transaction from the conflict transaction queue based on the transaction ID. After that, it clears the transaction event log of the completed transaction from the cache. If the completed transaction is eventually sent to multiple loading components, it is necessary to wait for all loading components to finish executing before the transaction event log of the completed transaction can be cleared from the cache.

[0220] like Figure 20 As shown, to ensure the atomicity of transactions, transactions are bound to database connections. All events within the same transaction are sent sequentially to the destination database for execution via the same database connection object until a commit event occurs. The loading component maintains a loading mapping structure (txId-to-dbconn-map) from transaction IDs (Trxid) to database objects. The loading and dispatching thread reads target data from the target message queue, allocates it to a database connection object, and then places the database connection object into the loading ready queue. The execution loading thread retrieves a database connection object from the loading ready queue and sends the target data to the destination database for execution via the database connection object.

[0221] The workflow of the loading and dispatching thread is described as follows:

[0222] Step 1: The loading and dispatching thread reads the transaction event log from the transformation message queue. If the log is not found, it blocks and waits. Once the transaction event log is retrieved, it executes (Step 2).

[0223] Step 2: The load dispatch thread uses the transaction ID obtained from the transaction event log to query the load mapping structure from the transaction ID to the data connection object to see if a database connection object (dbconn) has been allocated. If it has been allocated, proceed to (Step 3); if it has not been allocated, proceed to (Step 4).

[0224] Step 3: The loading and dispatching thread sends the transaction event log data to the database connection object, then checks whether the allocated database connection is waiting for the transaction event log. If it is waiting, the data connection object is placed in the "ready queue" and then proceeds to (Step 1); if it is not waiting, proceed directly to (Step 1).

[0225] Step 4: The loading and dispatching thread retrieves a database connection object from the "idle queue". If it cannot retrieve one, it waits until it does. If an idle database connection object is retrieved, the transaction event log data is passed to the database connection object, and then (Step 5) is executed.

[0226] Step 5: The loading and dispatching thread puts the idle database connection objects with transaction event logs set into the "ready queue" and then proceeds to (Step 1);

[0227] The workflow of executing the loading thread is described as follows:

[0228] Step 7: The loading thread retrieves a database connection object from the loading ready queue. If no object is found, it blocks and waits until a connection is found. If a connection is found, proceed to (Step 8).

[0229] Step 8: The loading thread submits commands to the destination database one by one according to the transaction event log in the transaction, and executes the transaction. If the last item in the current transaction log is a commit event, the execution is completed. Then the transaction ID is fed back to the loading completion notification queue of the corresponding reading component, and then proceeds to (Step 9); if it is not a commit event, the execution is completed and then proceeds to (Step 10).

[0230] Step 9: The loading thread places the current database connection object into the idle queue, then proceeds to (Step 7).

[0231] Step 10: The loading thread is executed to put the current database connection object into the waiting connection queue, then proceed to (Step 7).

[0232] This invention, based on the real-time characteristics of transaction event log data synchronization and the concept of component-based functional modules, implements a real-time, efficient, and flexible data synchronization system. Real-time incremental data synchronization can be achieved based on transaction event logs, and the component-based functional modules support users in flexibly designing diverse data synchronization tasks. The three types of components internally support pipelined and concurrent execution design, greatly improving the efficiency of data conversion and synchronization by maximizing the use of computer hardware and software resources.

[0233] Example 2:

[0234] like Figure 21 The diagram shown is an architectural schematic of a component-based data synchronization device according to an embodiment of the present invention. The quantization evaluation device in this embodiment includes one or more processors 31 and a memory 32. Figure 21 Take a processor 31 as an example.

[0235] Processor 31 and memory 32 can be connected via a bus or other means. Figure 21 Taking the example of a connection between China and Israel via a bus.

[0236] The memory 32, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the componentized data synchronization method in Embodiment 1. The processor 31 executes the componentized data synchronization method by running the non-volatile software programs and instructions stored in the memory 32.

[0237] Memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 32 may optionally include memory remotely located relative to processor 31, which can be connected to processor 31 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0238] The program instructions / modules are stored in the memory 32. When executed by one or more processors 31, they perform the component-based data synchronization method described in Embodiment 1 above, for example, the method described above. Figure 5 , Figure 7 , Figures 12-14 , Figure 16 and Figures 18-19 The steps shown.

[0239] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0240] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0241] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A component-based data synchronization method, characterized in that, include: The required data synchronization task is obtained by combining at least one read component and at least one load component. Alternatively, at least one reading component, at least one transformation component, and at least one loading component can be combined to obtain the required data synchronization task; The system retrieves transaction event logs belonging to the same transaction from the reading component and stores these logs in a corresponding preset cache set. These transaction event logs can be all or a portion of consecutive transaction event logs for the transaction. When a transaction event log is a commit event, the corresponding log is added from the preset cache set to a ready-to-transmit queue, resulting in a ready transaction. Depending on whether there are concurrent conflicts among the ready transactions in the ready-to-transmit queue, the corresponding ready transaction is selectively transmitted to all subsequent components of the reading component; or the corresponding ready transaction is added to a conflicting transaction queue until the transactions that conflict with the ready transaction are completed. Then, the ready transaction is removed from the conflicting transaction queue and transmitted to all subsequent components. After the transaction event logs of the current transaction are processed, the transaction event logs of the next transaction are retrieved until a stop-retrieval instruction is received. The subsequent components are either conversion components or loading components. When the subsequent component of the reading component is a transformation component, the transaction event log is transformed by at least one transformation component to obtain the target data; the target data is then transferred to the loading component; and the target data is then stored in the database by the loading component. When the subsequent component of the reading component is a loading component, the target data is loaded into the database through the loading component.

2. The component-based data synchronization method according to claim 1, characterized in that, The method involves selectively transmitting the corresponding ready transactions to all subsequent components based on whether there are concurrent conflicts among the ready transactions in the transmission ready queue. Alternatively, the corresponding ready transactions can be added to the conflicting transaction queue, including: Sequentially determine whether there are any transaction event logs in the execution transaction queue that conflict with the current ready transaction; when there is a conflict, record all transaction event logs that conflict with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue. Sequentially determine whether there are any transaction event logs in the conflict transaction queue that conflict with the current ready transaction; when there is a conflict, record all transaction event logs that conflict with the current ready transaction in the conflict transaction queue, and attach the current ready transaction to the conflict transaction queue. When there are no concurrent conflicts in the current ready transaction, the current ready transaction is added to the execution transaction queue, and the ready transaction is transmitted to all subsequent components according to the execution transaction queue.

3. The component-based data synchronization method according to claim 1, characterized in that, After the transaction event log of the current transaction is processed, the transaction event log of the next transaction is retrieved, until a stop retrieval instruction is received, including: Retrieve completed transactions from the loading completion notification queue, and determine whether all loading components that have transmitted the same transaction event log as the current completed transaction have completed execution; wherein, the current transaction is considered a completed transaction after its transaction event log has been processed. When all loading components have been executed, clear the preset cache set corresponding to the current completed transaction; obtain at least one ready transaction that has a concurrent conflict with the current completed transaction from the conflict transaction queue, and release the concurrency constraint between each ready transaction and the current completed transaction; Determine whether the acquired ready transactions have other concurrency constraints, add ready transactions without concurrency constraints to the execution transaction queue, and transmit the corresponding ready transactions to all subsequent components according to the execution transaction queue; Remove the currently completed transaction from the execution transaction queue, retrieve the transaction event log of the next transaction, and continue until a stop retrieval instruction is received.

4. The component-based data synchronization method according to claim 1, characterized in that, The transformation component includes a transformation dispatch thread and an execution transformation thread; wherein, one transformation component corresponds to one transformation dispatch thread, and one transformation component corresponds to at least one execution transformation thread; the transformation of the transaction event log through at least one transformation component to obtain target data includes: The transaction event log is obtained by transforming the dispatch thread. In the transformation mapping structure, it is queried whether the transaction corresponding to the transaction event log has been assigned to an execution transformation thread. If not assigned, an execution transformation thread is assigned to the transaction through the transformation dispatch thread, and the transaction and the corresponding execution transformation thread are written into the transformation mapping structure; wherein, one execution transformation thread processes all transaction event logs of a transaction; If already allocated, the transaction event log is passed to the corresponding execution transformation thread via the transformation dispatch thread; The allocated transaction event logs are processed by the conversion thread. When the transaction event logs match the user conversion rules, the corresponding conversion operations are performed on the transaction event logs to obtain the target data.

5. The component-based data synchronization method according to claim 1, characterized in that, The step of transmitting the target data to the loading component includes: If the number of concurrent transactions exceeds a preset threshold, the target data corresponding to all transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread, or the target data corresponding to the current continuous transaction event logs of the transaction is transmitted to the target message queue by executing a conversion thread; wherein, one transaction corresponds to at least one transaction event log; If the number of concurrent transactions is less than or equal to a preset threshold, the target data corresponding to the preset number of transaction event logs will be transmitted to the target message queue by executing the conversion thread. Specifically, after the target data corresponding to the transaction's commit event has been transmitted, the transaction is deleted from the transformation mapping structure.

6. The component-based data synchronization method according to claim 1, characterized in that, The loading component includes a loading and dispatch thread and an execution loading thread; wherein, one loading component corresponds to one loading and dispatch thread, and one loading component corresponds to at least one execution loading thread; the step of loading the target data into the database through the loading component includes: The loading and distribution thread reads the target data corresponding to the transaction event log from the target message queue, and queries the loading mapping structure to see if the transaction corresponding to the target data has been allocated a database connection object. If not allocated, when a free database connection object exists, the free database connection object is allocated to the transaction and recorded in the load mapping structure; the free database connection object is placed in the load ready queue, and the target data is read from the target message queue again until the target message queue is empty; wherein, one execution load thread processes all transaction event logs for one transaction; If the target data has been allocated, the target data is passed to the corresponding database connection object. When the database connection object is waiting for the transaction event log, the database connection object is placed in the loading ready queue, and the target data is read from the target message queue again. When the database connection object is not waiting for the transaction event log, the target data is read directly from the target message queue again until the target message queue is empty. The loading thread retrieves database connection objects from the loading ready queue and loads the corresponding target data into the database in the order of the transaction event logs within the transaction.

7. The component-based data synchronization method according to claim 6, characterized in that, The step of retrieving database connection objects from the loading-ready queue by executing the loading thread and sequentially loading the corresponding target data into the database according to the order of transaction event logs in the transaction includes: If the last target data is not a commit event, after the data is inserted into the database, the current database connection object is placed in the waiting connection queue, and the database connection object is retrieved from the loading ready queue again until the loading ready queue is empty. If the last target data is a commit event, and the loading component is the last loading component, then the transaction is added to the loading completion notification queue of the corresponding reading component; if the waiting connection queue is empty, then the current database connection object is made idle; if the waiting connection queue is not empty, then the current database connection object is assigned to the head transaction in the waiting connection queue, and the assigned database connection object is placed in the loading ready queue, and database connection objects are retrieved from the loading ready queue again until the loading ready queue is empty.

8. The component-based data synchronization method according to claim 1, characterized in that, The process involves combining at least one reading component and at least one loading component to obtain the required data synchronization task. Alternatively, combining at least one read component, at least one transformation component, and at least one load component yields the desired data synchronization task, including: The connection relationship between at least one read component and at least one load component is determined based on the processing logic of the transaction event log; or, the connection relationship between at least one read component, at least one transformation component, and at least one load component is determined based on the processing logic of the transaction event log. Based on the connection relationship, the corresponding components are connected together using directed arrow lines to obtain the data synchronization task; Among them, a reading component can be connected to at least one conversion component and / or at least one loading component, a conversion component can be connected to at least one conversion component and / or at least one loading component, and at least one of the loading components can be connected to the destination database.

9. A modular data synchronization device, characterized in that, It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. When executed by the processor, the instructions are used to implement the componentized data synchronization method according to any one of claims 1-8.

Images