Method and system for parallel synchronization of incremental data

Through the parallel synchronization incremental data method, the thread pool and queue mechanism are used to process incremental data, the problem of slow incremental data synchronization speed during database migration is solved, real-time data synchronization and transaction integrity are achieved, and business convergence needs are met. .

CN118939724APending Publication Date: 2024-11-12BEIJING VASTDATA TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410914706.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-01
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During the database migration process of the prior art, incremental data synchronization speed is slow, which cannot meet the needs of business parallel operation, and the serial writing method makes it difficult to ensure data consistency and integrity.

Method used

Using the method of synchronizing incremental data in parallel, the core thread pool, subquery and current queue is initialized by the incremental main thread, the incremental data is processed according to the queue length limit and data conflict situations, and the data writing is carried out using the subthreads of the concurrent execution thread pool. , reorder data tables to improve synchronization efficiency.

Benefits of technology

Improve incremental synchronization speed, reduce data delay between the target and source, ensure the order and integrity of transaction submissions, and realize real-time data synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939724A_ABST
    Figure CN118939724A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data synchronization, and provides a method and system for parallel synchronization of incremental data, and the method comprises the steps: employing an incremental main thread to initialize a core thread pool, a sub-queue and a current queue; and processing the collected incremental data according to whether the current queue reaches the length limit or not and whether data conflict exists between the currently processed transaction and the data of the sub-queue or not. According to the method and the system for parallel synchronization of the incremental data, the incremental synchronization speed can be increased, the data delay of the target end and the source end can be reduced, the sequence and the integrity of transaction submission can be ensured, and the effect of real-time synchronization of the data can be achieved through a mode of concurrently writing the data and reordering the data in the subqueues.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method and system for parallel synchronization of incremental data. Background Art

[0002] With the improvement of domestic database technology and products, more and more institutions or enterprises are turning to the domestic database openGauss. In the process of replacing the database, the data of the source database needs to be completely migrated to the new database. The business side requires that the migration of the database does not affect the normal writing and query of business data. Therefore, the incremental data generated during the full migration process also needs to be synchronized to the target database as soon as possible, so as to align the data in the databases on both sides in the shortest possible time to facilitate business cutover and switching. In addition, some databases may have large incremental data. In the process of incremental synchronization, the data needs to be written to the target database as real-time as possible to meet the needs of business parallel operation.

[0003] In actual applications, in order to ensure the order and integrity of transaction writing, the collected incremental data needs to be written to the target database in a serial manner to ensure the consistency and integrity of the data in the target database. That is, when querying the target database at any time, all the snapshot data that has occurred in the source database will be used, and when data anomalies occur, incomplete transactions will not be written and committed to the target database. However, writing to the target database in a serial manner will result in slow synchronization speed, and it will not be possible to complete all incremental data synchronization during the business cutover window period; and due to the slow synchronization speed, the data difference between the source database and the target database is large, and it is also impossible to meet the requirements of business parallel operation.

[0004] Therefore, how to provide an incremental data synchronization method to synchronously and parallelly write the collected incremental data into the target database while ensuring the order and integrity of transactions has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, in order to overcome the deficiencies of the prior art, the present invention aims to provide a method and system for parallel synchronization of incremental data.

[0006] According to a first aspect of the present invention, a method for synchronizing incremental data in parallel is provided, the method comprising:

[0007] Use incremental main thread to initialize core thread pool, sub-queue and current queue;

[0008] Process the collected incremental data based on whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue.

[0009] Preferably, in the method for synchronizing incremental data in parallel of the present invention, the incremental main thread is used to initialize the core thread pool, the sub-queue and the current queue, including:

[0010] Use the incremental main thread to set the number of core threads in the thread pool, initialize the number of sub-queues according to the set number of core threads, and set a length limit for each sub-queue;

[0011] Use the incremental main thread to initialize a buffer queue as the current queue and set a length limit for the current queue.

[0012] Preferably, in the method for parallel synchronization of incremental data of the present invention, the length limit of the current queue is the same as the length limit of the sub-queue.

[0013] Preferably, in the method for synchronizing incremental data in parallel of the present invention, processing the collected incremental data according to whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue includes:

[0014] Determine whether the current queue has reached the length limit. If the current queue has not reached the length limit, determine whether the DML data of the currently processed transaction conflicts with the data of the subqueue.

[0015] If the DML data of the currently processed transaction conflicts with the data of the sub-queue, the data of all sub-queues will be written to the database through the sub-threads of the concurrent execution thread pool, the currently processed transaction will be written to the current queue, and the transaction information of the current queue will be updated;

[0016] If there is no conflict between the DML data of the currently processed transaction and the data of the sub-queue, the currently processed transaction is written to the current queue and the transaction information of the current queue is updated.

[0017] Preferably, in the method for synchronizing incremental data in parallel of the present invention, determining whether there is a conflict between the DML data of the currently processed transaction and the data of the subqueue includes:

[0018] According to the transaction information cache corresponding to the currently processed transaction and the sub-queue data, determine whether the DML data of the currently processed transaction and the sub-queue data have the same unique identifier of the data table and data row affected by the DML;

[0019] If the DML data of the currently processed transaction and the data of the subqueue have the same unique identifier of the data table and data row acted upon by the DML, it is determined that there is a data conflict between the DML data of the currently processed transaction and the data of the subqueue;

[0020] If the DML data of the currently processed transaction and the data of the subqueue do not have the same unique identifier of the data table and data row acted upon by the DML, it is determined that there is no data conflict between the DML data of the currently processed transaction and the data of the subqueue.

[0021] Preferably, in the method for parallel synchronization of incremental data of the present invention, the unique identifiers of the data table and data row acted upon by the DML data of the currently processed transaction and the DML corresponding to the data of the sub-queue are respectively recorded in the transaction information cache corresponding to the currently processed transaction and the data of the sub-queue. For a data table containing a primary key, a unique key, or a foreign key, the data table name and the column name corresponding to the primary key column, the unique key column, or the foreign key column and the above-mentioned column value are used as the unique identifier; for a table not containing a primary key, a unique key, or a foreign key, the data table name and a hash value calculated based on the column values ​​of all columns are used as the unique identifier.

[0022] Preferably, in the method for synchronizing incremental data in parallel of the present invention, processing the collected incremental data according to whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue includes:

[0023] If the current queue reaches the length limit, determine whether the sum of the number of remaining subqueues and the number of current queues reaches the subqueue number limit;

[0024] If the sum of the number of remaining sub-queues and the number of current queues does not reach the sub-queue number limit, the data of the current queue is written into the sub-queues in the order of the sub-queues, the transaction information of the current queue is written into the transaction information cache of the sub-queues, the current queue and the transaction information of the current queue are cleared, and the current transaction is judged again based on the sub-queue transaction cache information whether there is a data conflict between the currently processed transaction and the sub-queue data.

[0025] Preferably, in the method for parallel synchronization of incremental data of the present invention, the collected incremental data is processed according to whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the sub-queue, including: if the sum of the number of remaining sub-queues and the current number of queues reaches the sub-queue number limit, the current queue is written to the last sub-queue, the transaction information of the current queue and sub-queues is cleared, and the data of all sub-queues are written to the database through the sub-threads of the concurrent execution thread pool.

[0026] Preferably, in the method for synchronizing incremental data in parallel of the present invention, writing data of all sub-queues into a database by concurrently executing sub-threads of a thread pool includes:

[0027] When the data of the sub-queue is stored in the database through the sub-thread of the concurrent execution thread pool, it is determined whether the data table acted on by the DML command in the sub-queue contains a foreign key table;

[0028] When the data table acted on by the DML command in the subqueue does not contain a foreign key table, the data in the subqueue is rearranged according to the table name, and the data in the subqueue is written to the database in the rearranged order through the PBE method; when the data table acted on by the DML command in the subqueue contains a foreign key table, the data in the subqueue is written to the database in the default order through the PBE method;

[0029] Submit the transactions of the subqueues in the order of the subqueues. After the transactions of the subqueues are submitted, clear the subqueues and the cache data corresponding to the transaction information of the subqueues.

[0030] According to a second aspect of the present invention, a system for parallel synchronization of incremental data is provided, the system comprising a parallel synchronization server, the parallel synchronization server being used to: initialize a core thread pool, a sub-queue and a current queue using an incremental main thread; and process the collected incremental data according to whether the current queue has reached a length limit and whether there is a data conflict between a currently processed transaction and the data of the sub-queue.

[0031] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect of the present invention when executing the program.

[0032] The method and system for parallel synchronization of incremental data of the present invention can improve the speed of incremental synchronization, reduce data delay between the target end and the source end, and ensure the order and integrity of transaction submission, thereby achieving the effect of real-time data synchronization, by concurrently writing data and reordering data in sub-queues. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a system for a method for parallel synchronization of incremental data applicable to an embodiment of the present application;

[0035] Figure 2 is an execution flow chart of a method for synchronizing incremental data in parallel according to an embodiment of the present application;

[0036] Figure 3 This is an example diagram of a process of storing data of a sub-queue into a database by concurrently executing a sub-thread of a thread pool in a method for synchronizing incremental data in parallel according to an embodiment of the present invention;

[0037] Figure 4 A principle example diagram of a method for synchronizing incremental data in parallel according to an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the structure of the device provided by the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0041] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0042] Figure 1 An exemplary system for a method for parallel synchronization of incremental data applicable to an embodiment of the present application is shown. Figure 1 As shown, the system may include a parallel synchronization server 101, a communication network 102 and / or one or more parallel synchronization clients 103. Figure 1 The example in FIG. 1 is a plurality of parallel synchronization clients 103 .

[0043] The parallel synchronization server 101 can be any appropriate server for storing information, data, programs and / or any other suitable type of content. In some embodiments, the parallel synchronization server 101 can perform appropriate functions. For example, in some embodiments, the parallel synchronization server 101 can be used to synchronize incremental data in parallel. As an optional example, in some embodiments, the parallel synchronization server 101 can be used to initialize the core thread pool, sub-queues and current queue using an incremental main thread; process the collected incremental data based on whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the sub-queue.

[0044] As another example, in some embodiments, the parallel synchronization server 101 may send a method for synchronizing incremental data in parallel to the parallel synchronization client 103 for the user to use according to the request of the parallel synchronization client 103 .

[0045] As an optional example, in some embodiments, the parallel synchronization client 103 is used to provide a visual parallel synchronization interface, which is used to receive a user's selection input operation for parallel synchronization incremental data, and, in response to the selection input operation, obtain a parallel synchronization interface corresponding to the option selected by the selection input operation from the parallel synchronization server 101 and display the parallel synchronization interface, wherein the parallel synchronization interface at least displays information on the parallel synchronization incremental data and operation options for the information on the parallel synchronization incremental data.

[0046] In some embodiments, the communication network 102 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 102 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The parallel synchronization client 103 can be connected to the communication network 102 via one or more communication links (e.g., communication link 104), and the communication network 102 can be linked to the parallel synchronization server 101 via one or more communication links (e.g., communication link 105). The communication link can be any communication link suitable for transmitting data between the parallel synchronization client 103 and the parallel synchronization server 101, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0047] The parallel synchronization client 103 may include any one or more clients that present an interface related to the parallel synchronization incremental data in an appropriate form for use and operation by the user. In some embodiments, the parallel synchronization client 103 may include any suitable type of device. For example, in some embodiments, the parallel synchronization client 103 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable type of client device.

[0048] Although the parallel synchronization server 101 is illustrated as one device, in some embodiments, any suitable number of devices may be used to perform the functions performed by the parallel synchronization server 101. For example, in some embodiments, multiple devices may be used to implement the functions performed by the parallel synchronization server 101. Alternatively, the functions of the parallel synchronization server 101 may be implemented using a cloud service.

[0049] Based on the above system, an embodiment of the present application provides a method for parallel synchronization of incremental data, which is illustrated by the following embodiment.

[0050] Figure 2 The present invention is a flowchart of an execution of a method for parallel synchronization of incremental data according to an embodiment of the present application.

[0051] The method for synchronizing incremental data in parallel in this embodiment can be executed on a parallel synchronization server, and the method for synchronizing incremental data in parallel includes the following steps:

[0052] Use incremental main thread to initialize the core thread pool, sub-queue and current queue.

[0053] As an optional example, this embodiment uses an incremental main thread to set the number of core threads in the thread pool, initializes the number of sub-queues according to the set number of core threads, and sets a length limit for each sub-queue; uses an incremental main thread to initialize a buffer queue as the current queue, and sets a length limit for the current queue. It should be noted that in the method of this embodiment, the length limit of the current queue is the same as the length limit of the sub-queue. In the method of this embodiment, the number of core threads in the thread pool is set according to the number of concurrency, and the number and length of the sub-queues are set according to the number of concurrency.

[0054] After completing the initialization of the core thread pool, the subqueue, and the current queue, the method of this embodiment processes the collected incremental data according to whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue.

[0055] As an optional example, in the method of this embodiment, it is determined whether the current queue has reached the length limit. If the current queue has not reached the length limit, it is determined whether there is a conflict between the DML data of the currently processed transaction and the data of the sub-queue. It should be noted that in the method of this embodiment, based on the transaction information cache corresponding to the currently processed transaction and the data of the sub-queue, it is determined whether the DML data of the currently processed transaction and the data of the sub-queue have the same unique identifier of the data table and data row affected by the DML; if the DML data of the currently processed transaction and the data of the sub-queue have the same unique identifier of the data table and data row affected by the DML, it is determined that there is a data conflict between the DML data of the currently processed transaction and the data of the sub-queue; if the DML data of the currently processed transaction and the data of the sub-queue do not have the same unique identifier of the data table and data row affected by the DML, it is determined that there is no data conflict between the DML data of the currently processed transaction and the data of the sub-queue. In the method of this embodiment, the unique identifiers of the data table and data row acted upon by the DML data of the currently processed transaction and the DML corresponding to the data of the subqueue are respectively recorded in the transaction information cache corresponding to the currently processed transaction and the data of the subqueue. For a data table containing a primary key, a unique key, or a foreign key, the data table name and the column name corresponding to the primary key column, the unique key column, or the foreign key column and the above column value are used as the unique identifier; for a table without a primary key, a unique key, or a foreign key, the data table name and a hash value calculated based on the column values ​​of all columns are used as the unique identifier.

[0056] It should be noted that in the method of this embodiment, the data of the subqueue refers to the DML operation data of all transactions in the subqueue. DML refers to Data Manipulation Language, which is a programming statement used for database operations and access operations on objects and data in the database. It is usually a subset of database-specific programming languages. In the SQL language, which is the standard for the information software industry, DML is centered on three instructions: INSERT, UPDATE, and DELETE, which represent insert, update, and delete, respectively.

[0057] If the DML data of the currently processed transaction conflicts with the data of the sub-queue, the data of all sub-queues are written to the database through the sub-threads of the concurrent execution thread pool, the currently processed transaction is written to the current queue, and the transaction information of the current queue is updated.

[0058] If there is no conflict between the DML data of the currently processed transaction and the data of the sub-queue, the currently processed transaction is written to the current queue and the transaction information of the current queue is updated.

[0059] If the current queue reaches the length limit, determine whether the sum of the number of remaining subqueues and the number of current queues reaches the subqueue number limit;

[0060] If the sum of the number of remaining sub-queues and the number of current queues does not reach the sub-queue number limit, the data of the current queue is written into the sub-queues in the order of the sub-queues, the transaction information of the current queue is written into the transaction information cache of the sub-queues, the current queue and the transaction information of the current queue are cleared, and the current transaction is judged again based on the sub-queue transaction cache information whether there is a data conflict between the currently processed transaction and the sub-queue data.

[0061] If the sum of the number of remaining subqueues and the current number of queues reaches the subqueue number limit, the current queue is written to the last subqueue, the transaction information of the current queue and subqueues is cleared, and the data of all subqueues is written to the database through the subthreads of the concurrent execution thread pool. Figure 3 FIG. 1 is a flowchart of storing data of a sub-queue by concurrently executing a sub-thread of a thread pool in a method for synchronizing incremental data in parallel according to an embodiment of the present invention; Figure 2 and Figure 3 As shown, in the method of this embodiment, when the data of the subqueue is stored in the database through the subthread of the concurrent execution thread pool, it is determined whether the data table acted on by the DML command in the subqueue contains a foreign key table; when the data table acted on by the DML command in the subqueue does not contain a foreign key table, the data in the subqueue is rearranged according to the table name, and the data in the subqueue is written into the database in the rearranged order through the PBE method; when the data table acted on by the DML command in the subqueue contains a foreign key table, the data in the subqueue is written into the database in the default order through the PBE method; the transactions of the subqueue are submitted in the order of the subqueues, and after the transaction submission of the subqueue is completed, the cached data corresponding to the transaction information of the subqueue and the subqueue are cleared. In the method of this embodiment, PBE is a message-based communication protocol used by the PostgreSQL database, which allows effective data exchange between the client and the server. The PBE protocol optimizes the data packaging and unpacking process, reduces the overhead in data transmission, and thus improves the efficiency of data processing.

[0062] The method for parallel synchronization of incremental data of this embodiment is described in detail below in a specific application scenario.

[0063] First, the incremental main thread initializes the thread pool at the startup stage, sets the number of core threads in the thread pool to N, the number of subqueues to N, and sets the length limit of each subqueue. Initialize a cache queue as the current queue, and set the length limit of the current queue to be the same as the length limit of the subqueue.

[0064] Then, start processing the collected incremental data according to the transaction dimension: determine whether the current queue has reached the length limit, that is, whether the current queue is full. When processing the first transaction, the current queue is not full. The following explains the situation where there may be data in both the current queue and the sub-queue after executing multiple processing transactions.

[0065] If the current queue is not full, determine whether there is a data conflict between the currently processed transaction and the data of the sub-queue, which can be determined based on the sub-queue transaction information cache. If there is a data conflict, the data of all sub-queues will be stored in the database. This process uses the sub-threads of the thread pool to execute concurrently. In the process of executing each queue storage, determine whether the table in the queue contains a foreign key table. If not, rearrange the data in the queue according to the table name; if it contains a foreign key table, follow the original order. Then use the pbe method to write to the database, and submit the sub-queue transaction in the order of the sub-queues. After the submission is completed, clear the sub-queue and the transaction information cache data of the sub-queue. The rearrangement method can make full use of the performance advantages of pbe, improve the speed of data writing, and thus improve the performance of overall data increment.

[0066] After the warehousing process is completed, the current transaction is executed into the current queue and the transaction information of the current queue is updated; if there is no data conflict, the current transaction is directly written into the current queue and the transaction information of the current queue is updated. The transaction information data needs to record the key transaction information of each DML in the transaction. For tables with primary keys / unique keys / foreign keys, the affected schema name + table name + key corresponding column name and corresponding column value are recorded respectively. For tables without primary keys, the affected schema name + table name and the corresponding hash value of all column values ​​of the affected rows are recorded.

[0067] If the current queue is full, determine whether the number of subqueues is less than N-1, that is, determine whether the current number of subqueues + the current queue reaches the subqueue number limit. If the number of subqueues is less than N-1, write the data in the current queue to the subqueue in the order of the subqueues, that is, write the first queue first, then the second queue, until the N-2 queue, and write the transaction information of the current queue to the transaction information cache of the subqueue, and then clear the transaction information of the current queue and the current queue. Then re-determine whether there is a data conflict between the currently processed transaction and the data of the subqueue.

[0068] If the number of sub-queues is greater than or equal to N-1, the current queue is written into the sub-queue in the order of the sub-queues, that is, the current queue is written into the last queue in the sub-queue, and then the transaction information of the current queue and sub-queues is cleared, and the storage process is executed. After the storage process is completed, the current transaction is executed into the current queue and the transaction information of the current queue is updated.

[0069] Figure 4FIG. 1 is a schematic diagram showing a method for synchronizing incremental data in parallel according to an embodiment of the present invention. Figure 4 As shown, the method and system for parallel synchronization of incremental data of this embodiment establish conflict judgment basis for whether transactions can be written to the target library in parallel. By recording the unique mark of the row affected by the transaction, it is judged whether different transactions affect the data of the same row. If the same row is not affected, it is considered that there is no conflict between the transactions and they can be written to the target library in parallel. That is, if there is a conflict between transactions, the data in the previous sub-queue is written in parallel and submitted, and the conflicting transaction data enters the next round of transaction conflict detection and concurrent writing process. The main thread writes the data without conflict between transactions into the queue of the sub-thread in batches according to the benchmark number and on the premise of ensuring the integrity of the transaction. The sub-thread determines whether the data in its queue can be reordered. When there is no foreign key constraint in the table in the queue, the data in the queue can be reordered according to the table, and then the data is written to the target library by PBE. After judging that the sub-thread has completed writing, the main thread performs transaction commit actions on the sub-thread in order to ensure that the transaction submission order is consistent with the order in which the source transaction occurs.

[0070] The method and system for parallel synchronization of incremental data in this embodiment can improve the speed of incremental synchronization, reduce data latency between the target and source ends, and ensure the order and integrity of transaction submission, thereby achieving real-time data synchronization, by concurrently writing data and reordering data in sub-queues.

[0071] like Figure 5 As shown, the present invention also provides a device, including a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 implements the above-mentioned method of parallel synchronization of incremental data by running the executable computer program.

[0072] Among them, the computer program in the memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0073] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected based on actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0075] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for parallel synchronization of incremental data, characterized in that: The method comprises: Use incremental main thread to initialize core thread pool, sub-queue and current queue; Process the collected incremental data based on whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue.

2. The method for parallel synchronization of incremental data according to claim 1, characterized in that: Use the incremental main thread to initialize the core thread pool, sub-queues, and current queue, including: Use the incremental main thread to set the number of core threads in the thread pool, initialize the number of sub-queues according to the set number of core threads, and set a length limit for each sub-queue; Use the incremental main thread to initialize a buffer queue as the current queue and set a length limit for the current queue.

3. The method for parallel synchronization of incremental data according to claim 2, characterized in that: The length limit of the current queue is the same as the length limit of the subqueue.

4. The method for parallel synchronization of incremental data according to claim 1, characterized in that: Process the collected incremental data based on whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue, including: Determine whether the current queue has reached the length limit. If the current queue has not reached the length limit, determine whether the DML data of the currently processed transaction conflicts with the data of the subqueue. If the DML data of the currently processed transaction conflicts with the data of the sub-queue, the data of all sub-queues will be written to the database through the sub-threads of the concurrent execution thread pool, the currently processed transaction will be written to the current queue, and the transaction information of the current queue will be updated; If there is no conflict between the DML data of the currently processed transaction and the data of the sub-queue, the currently processed transaction is written to the current queue and the transaction information of the current queue is updated.

5. The method for parallel synchronization of incremental data according to claim 4, characterized in that: Determine whether the DML data of the currently processed transaction conflicts with the data of the subqueue, including: According to the transaction information cache corresponding to the currently processed transaction and the sub-queue data, determine whether the DML data of the currently processed transaction and the sub-queue data have the same unique identifier of the data table and data row affected by the DML; If the DML data of the currently processed transaction and the data of the subqueue have the same unique identifier of the data table and data row acted upon by the DML, it is determined that there is a data conflict between the DML data of the currently processed transaction and the data of the subqueue; If the DML data of the currently processed transaction and the data of the subqueue do not have the same unique identifier of the data table and data row acted upon by the DML, it is determined that there is no data conflict between the DML data of the currently processed transaction and the data of the subqueue.

6. The method for parallel synchronization of incremental data according to claim 5, characterized in that: The unique identifiers of the data table and data row affected by the DML data of the currently processed transaction and the DML corresponding to the data of the subqueue are recorded in the transaction information cache corresponding to the currently processed transaction and the data of the subqueue, respectively. For a data table with a primary key, unique key or foreign key, the data table name and the corresponding column name of the primary key column, unique key column or foreign key column and the above column value are used as the unique identifier; for a table without a primary key, unique key or foreign key, the data table name and the hash value calculated based on the column values ​​of all columns are used as the unique identifier.

7. The method for parallel synchronization of incremental data according to claim 4, characterized in that: Process the collected incremental data based on whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the subqueue, including: If the current queue reaches the length limit, determine whether the sum of the number of remaining subqueues and the number of current queues reaches the subqueue number limit; If the sum of the number of remaining sub-queues and the number of current queues does not reach the sub-queue number limit, the data of the current queue is written into the sub-queues in the order of the sub-queues, the transaction information of the current queue is written into the transaction information cache of the sub-queues, the current queue and the transaction information of the current queue are cleared, and the current transaction is judged again based on the sub-queue transaction cache information whether there is a data conflict between the currently processed transaction and the sub-queue data.

8. The method for parallel synchronization of incremental data according to claim 7, characterized in that: According to whether the current queue has reached the length limit and whether there is a data conflict between the currently processed transaction and the data of the sub-queue, the collected incremental data is processed, including: if the sum of the number of remaining sub-queues and the number of current queues reaches the sub-queue number limit, the current queue is written to the last sub-queue, the transaction information of the current queue and sub-queues is cleared, and the data of all sub-queues is written to the database through the sub-threads of the concurrent execution thread pool.

9. The method for parallel synchronization of incremental data according to claim 8, characterized in that: The data of all sub-queues are written to the database through the concurrent execution of the sub-threads of the thread pool, including: When the data of the sub-queue is stored in the database through the sub-thread of the concurrent execution thread pool, it is determined whether the data table acted on by the DML command in the sub-queue contains a foreign key table; When the data table acted on by the DML command in the subqueue does not contain a foreign key table, the data in the subqueue is rearranged according to the table name, and the data in the subqueue is written to the database in the rearranged order through the PBE method; when the data table acted on by the DML command in the subqueue contains a foreign key table, the data in the subqueue is written to the database in the default order through the PBE method; Submit the transactions of the subqueues in the order of the subqueues. After the transactions of the subqueues are submitted, clear the subqueues and the cache data corresponding to the transaction information of the subqueues.

10. A system for parallel synchronization of incremental data, characterized in that: The system includes a parallel synchronization server, which is used to: use an incremental main thread to initialize a core thread pool, a sub-queue and a current queue; and process the collected incremental data according to whether the current queue has reached a length limit and whether there is a data conflict between the currently processed transaction and the data of the sub-queue.