A parallel incremental synchronization method, system, storage medium and electronic device

Abstract

The application relates to a parallel incremental synchronization method, a system, a storage medium and an electronic device. The method comprises the following steps: establishing a materialized view log table for each data table to be synchronized; establishing a plurality of extraction threads, and extracting data in a plurality of materialized view log tables in batches and in parallel through the plurality of extraction threads, storing the data extracted in batches and in parallel in a plurality of data structures, and recording a starting sequence number and an ending sequence number of corresponding batch data in each data structure; caching the plurality of data structures in a plurality of buffer queues; establishing a plurality of warehousing threads, and warehousing the data structures cached in the plurality of buffer queues in parallel through the plurality of warehousing threads, confirming the incremental data to be synchronized according to the starting sequence number and the ending sequence number recorded in each data structure after warehousing, and deleting corresponding records in the materialized view log table. The application can synchronize data in a plurality of materialized view log tables in parallel.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of database synchronization, and particularly relates to a parallel incremental synchronization method and system, a storage medium and an electronic device. BACKGROUND

[0002] Under the needs of data backup and migration, localization and IOE removal, it is required to synchronize incremental data from an oracle source database to a target database, which can be an oracle or other types of databases. An oracle synchronization scheme based on a materialized view log table requires that one data table corresponds to one materialized view log table, and in order to ensure data consistency, extraction and warehousing need to be performed in sequence, and only one materialized view log table is extracted and warehoused at the same time. However, due to the influence of business or running environment, the warehousing performance of different tables is different, and therefore, if extraction and warehousing are performed in sequence, the efficiency is relatively low.

[0003] Therefore, there is an urgent need for a new incremental synchronization method to synchronize data of multiple materialized view log tables in parallel. SUMMARY

[0004] In view of the above problems in the prior art, the application provides a parallel incremental synchronization method, system, storage medium and electronic device, which can synchronize data of multiple materialized view log tables in parallel.

[0005] The application adopts the following technical scheme:

[0006] The application provides a parallel incremental synchronization method in the first aspect, which includes the following steps:

[0007] A materialized view log table is established for each data table to be synchronized;

[0008] Multiple extraction threads are established, and data in the multiple materialized view log tables is extracted in parallel and in batches through the multiple extraction threads, and the data extracted in parallel and in batches is stored in multiple data structures, and each data structure records a starting sequence number and an ending sequence number of the corresponding batch data;

[0009] The multiple data structures are cached in multiple buffer queues;

[0010] Multiple warehousing threads are established, and the data structures cached in the multiple buffer queues are warehoused in parallel through the multiple warehousing threads, and after warehousing, the incremental data for synchronization is confirmed according to the starting sequence number and the ending sequence number recorded by each data structure, and the corresponding record in the materialized view log table is deleted.

[0011] As a preferred solution, after the materialized view log table is established for each data table to be synchronized, the initial weight corresponding to each materialized view log table is given, and each materialized view log table is divided into a hot table and a normal table based on the initial weight corresponding to each materialized view log table. The plurality of extraction threads include a preset proportion of hot table data extraction threads and normal table data extraction threads, wherein the proportion of the hot table data extraction threads is greater than that of the normal table data extraction threads, the hot table data extraction threads preferentially extract data with a large weight in all hot tables, and the normal table data extraction threads preferentially extract data with a large weight in all normal tables.

[0012] As a preferred solution, the remaining synchronization data amount in each materialized view log table is further counted in real time, and the weight corresponding to each materialized view log table is adjusted according to the remaining synchronization data amount in each materialized view log table.

[0013] As a preferred solution, after the materialized view log table is established for each data table to be synchronized, each materialized view log table is further initialized and defined as a fast table or a slow table, the plurality of buffer queues include a fast table buffer queue and a slow table buffer queue, the data structure body corresponding to the fast table is cached to the fast table buffer queue, and the data structure body corresponding to the slow table is cached to the slow table buffer queue. The data synchronization speed of the fast table is greater than that of the slow table.

[0014] As a preferred solution, the real-time synchronization speed of each materialized view log table is further counted in real time to calculate the average real-time synchronization speed of all materialized view log tables, and the fast table and the slow table type of each materialized view log table are updated in real time based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all materialized view log tables.

[0015] As a preferred solution, the materialized view log table with a real-time synchronization speed greater than the average real-time synchronization speed of all materialized view log tables is defined as a fast table, and the materialized view log table with a real-time synchronization speed less than the average real-time synchronization speed of all materialized view log tables is defined as a slow table.

[0016] As a preferred solution, in the caching stage:

[0017] If the table corresponding to the extracted data structure body is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the data structure body is cached to the fast table buffer queue;

[0018] If the table corresponding to the extracted data structure body is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the extraction of the data in the table is suspended, and the extraction is continued after the data of the table in the slow table buffer queue is all warehoused;

[0019] If the table corresponding to the extracted data structure body is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, the data structure body is cached to the slow table buffer queue;

[0020] If the table corresponding to the extracted data structure body is currently defined as a slow table, and there is data corresponding to the table in the fast table buffer queue, the extraction of data in the table is suspended, and the extraction is continued after the data in the table in the fast table buffer queue is inserted into the slow table buffer queue.

[0021] As a preferred solution, in the warehousing stage:

[0022] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a slow table, the data structure body is inserted into the slow table buffer queue, and the warehousing of other data structure bodies in the fast table buffer queue is continued;

[0023] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the data structure body is inserted into the slow table buffer queue, and the warehousing of other data structure bodies in the fast table buffer queue is continued;

[0024] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the warehousing of the data structure bodies in the fast table buffer queue is continued;

[0025] The warehousing thread continuously warehouses the data structure bodies in the slow table buffer queue.

[0026] The second aspect of the embodiment of the application provides a parallel incremental synchronization system, which comprises, in sequence, a log table establishing module, an extraction module, a buffer module, a warehousing module, and an incremental data confirmation module, and the incremental data confirmation module is further connected with the log table establishing module;

[0027] The log table establishing module is used for establishing a materialized view log table for each data table to be synchronized.

[0028] The extraction module is used for establishing a plurality of extraction threads, and extracting data in a plurality of materialized view log tables in parallel and in batches through the plurality of extraction threads, and storing the data extracted in parallel and in batches in a plurality of data structure bodies respectively, each data structure body recording a starting sequence number and an ending sequence number of corresponding batch data.

[0029] The buffer module is used for caching the plurality of data structure bodies to a plurality of buffer queues respectively.

[0030] The warehousing module is used for establishing a plurality of warehousing threads, and warehousing the data structure bodies cached in the plurality of buffer queues in parallel through the plurality of warehousing threads.

[0031] An incremental data confirmation module is configured to confirm the incremental data according to the start sequence number and the end sequence number recorded by each data structure after the data structure is stored in the database, and delete the corresponding record in the materialized view log table.

[0032] As a preferred solution, the strategy module is further connected with the log table establishing module and the extraction module;

[0033] The strategy module is configured to assign an initial weight to each materialized view log table, and divide each materialized view log table into a hot table and a normal table based on the initial weight of each materialized view log table.

[0034] The extraction module establishes a plurality of extraction threads including hot table data extraction threads and normal table data extraction threads in a preset proportion, wherein the proportion of the hot table data extraction threads is greater than that of the normal table data extraction threads, and the hot table data extraction threads preferentially extract the data with a large weight in all hot tables, and the normal table data extraction threads preferentially extract the data with a large weight in all normal tables.

[0035] As a preferred solution, the strategy module is further configured to real-time count the remaining synchronization data amount in each materialized view log table, and adjust the weight corresponding to each materialized view log table according to the remaining synchronization data amount in each materialized view log table.

[0036] As a preferred solution, the strategy module is further connected with the buffer module.

[0037] The strategy module is further configured to initialize each materialized view log table as a fast table or a slow table, and the data synchronization speed of the fast table is greater than that of the slow table.

[0038] The plurality of buffer queues in the buffer module include a fast table buffer queue and a slow table buffer queue, and the data structure corresponding to the fast table is cached to the fast table buffer queue, and the data structure corresponding to the slow table is cached to the slow table buffer queue.

[0039] As a preferred solution, the strategy module is further configured to real-time count the real-time synchronization speed of each materialized view log table, calculate the average real-time synchronization speed of all materialized view log tables, and real-time update the fast table and the slow table of each materialized view log table based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all materialized view log tables.

[0040] As a preferred solution, the materialized view log table with a real-time synchronization speed greater than the average real-time synchronization speed of all materialized view log tables is defined as a fast table in the strategy module, and the materialized view log table with a real-time synchronization speed less than the average real-time synchronization speed of all materialized view log tables is defined as a slow table.

[0041] As a preferred solution, the buffer module buffers the data structure in the cache stage:

[0042] If the table corresponding to the extracted data structure is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the data structure is cached to the fast table buffer queue;

[0043] If the table corresponding to the extracted data structure is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the extraction of the data in the table is suspended, and the extraction is continued after the data in the table in the slow table buffer queue is all warehoused;

[0044] If the table corresponding to the extracted data structure is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, the data structure is cached to the slow table buffer queue;

[0045] If the table corresponding to the extracted data structure is currently defined as a slow table, and there is data corresponding to the table in the fast table buffer queue, the extraction of the data in the table is suspended, and the extraction is continued after the data in the table in the fast table buffer queue is all inserted into the slow table buffer queue.

[0046] As a preferred solution, the strategy module is further connected with the warehousing module, and the warehousing module is in the warehousing stage:

[0047] If the table corresponding to the data structure in the fast table buffer queue to be currently warehoused is currently defined as a slow table, the data structure is inserted into the slow table buffer queue, and the warehousing of other data structures in the fast table buffer queue is continued;

[0048] If the table corresponding to the data structure in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the data structure is inserted into the slow table buffer queue, and the warehousing of other data structures in the fast table buffer queue is continued;

[0049] If the table corresponding to the data structure in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the warehousing of the data structure in the fast table buffer queue is continued;

[0050] The warehousing thread continuously warehouses the data structure in the slow table buffer queue.

[0051] The third aspect of the embodiment of the application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, the computer instructions are used for making the computer execute the parallel incremental synchronization method as described in the first aspect and any one of the first aspect of the embodiment of the application.

[0052] The fourth aspect of the embodiment of the present application provides an electronic device, comprising a memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the parallel incremental synchronization method according to the first aspect of the embodiment of the present application and any one of the first aspect.

[0053] The present application has the following beneficial effects:

[0054] The present application provides a parallel extraction method and a parallel warehousing method based on a buffer queue pool and a sequence (sequence is a sequence number recorded in a materialized view log table) confirmation mechanism: the present application fully considers the characteristics of the materialized view log table, and in combination with the actual situation, serially extracts and warehouses the incremental data from the same materialized view log table, and parallelly extracts and warehouses the incremental data from different data tables. The synchronization performance is improved.

[0055] The present application provides a strategy of preferentially extracting a hot table and preferentially warehousing a fast table: the present application uses self-defined parameters and real-time running data to adjust the behaviors of extraction and warehousing in real time. The hot table data with high real-time requirements and frequent operations are preferentially extracted by using a weight mechanism; the real-time running data is used to distinguish between fast and slow tables, and the parallel warehousing strategy is adopted for different tables, thereby reducing the influence of the slow warehousing table on the overall synchronization performance, improving the throughput of the incremental synchronization, and reducing the delay. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 It is a flowchart of a parallel incremental synchronization method according to an embodiment of the present application;

[0058] Figure 2 It is a structural block diagram of a parallel incremental synchronization system according to an embodiment of the present application;

[0059] Figure 3 It is a schematic diagram of data extraction, data caching and data warehousing in the parallel incremental synchronization system according to the present application;

[0060] Figure 4 It is a structural schematic diagram of a computer readable storage medium according to the embodiment of the present application;

[0061] Figure 5 is a structural schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The present application is illustrated by way of specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure herein. The present application can also be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details herein based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0063] Embodiment One

[0064] Referring to Figure 1 , the present embodiment provides a parallel incremental synchronization method, including the steps of:

[0065] A materialized view log table is established for each data table to be synchronized, including the names, types, position indexes and other metadata of the primary key and all other columns, containing rowid, sequence, primary key, filter columns and including new values and other features.

[0066] A plurality of extraction threads are established to form an extraction thread pool, which can be referred to as Figure 2 , and the data in the plurality of materialized view log tables is extracted in parallel and in batches by the plurality of extraction threads, and the data extracted in parallel and in batches is stored in a plurality of DataRecord data structures, each of which records the starting sequence number sequence and the ending sequence number sequence of the corresponding batch of data, and sequence is the sequence number recorded in the materialized view log table;

[0067] The plurality of data structures are cached in a plurality of buffer queues, which can be referred to as Figure 2 , each table in the buffer queue represents a data structure, and a batch of data in a materialized view log table is stored in a data structure;

[0068] A plurality of warehousing threads are established to form a warehousing thread pool, which can be referred to as Figure 2 , and the data structures cached in the plurality of buffer queues are warehoused in parallel by the plurality of warehousing threads, and after warehousing, the incremental data for synchronization is confirmed according to the starting sequence number and the ending sequence number recorded in each data structure, and the corresponding records in the materialized view log table are deleted.

[0069] As can be seen, the application proposes a parallel extraction mode based on a "buffer queue pool" and a sequence confirmation mechanism and a parallel warehousing mode: the application fully considers the characteristics of the materialized view log table, and in combination with the actual situation, the incremental data derived from the same materialized view log table is serially extracted and warehoused, and the incremental data derived from different data tables is parallelly extracted and warehoused, thereby improving the synchronization performance. Figure 2 As shown in the figure, table A2 and table A1 are derived from the same materialized view log table, and table A1, table B1, table C1 and table D are derived from different materialized view log tables.

[0070] Specifically:

[0071] After establishing the materialized view log table for each data table to be synchronized, the initial weight corresponding to each materialized view log table is given, and each materialized view log table is divided into a hot table and a normal table based on the initial weight corresponding to each materialized view log table, and the plurality of extraction threads include a preset proportion of hot table data extraction threads and normal table data extraction threads, wherein the proportion of the hot table data extraction threads is greater than that of the normal table data extraction threads, for example, 3 hot table data extraction threads and 1 normal table data extraction thread are set. The hot table data extraction thread preferentially extracts the data with a large weight in all hot tables, and the normal table data extraction thread preferentially extracts the data with a large weight in all normal tables. It should be noted that the initial weight is artificially set, and a larger weight is set for the materialized view log table with high real-time synchronization requirements.

[0072] Further, the remaining synchronization data amount in each materialized view log table is further statistically calculated, and the weight corresponding to each materialized view log table is adjusted according to the remaining synchronization data amount in each materialized view log table, that is, the weight of the materialized view log table with a large remaining synchronization data amount is increased, and vice versa.

[0073] After establishing the materialized view log table for each data table to be synchronized, each materialized view log table is also initialized and defined as a fast table or a slow table, and the plurality of buffer queues include a fast table buffer queue and a slow table buffer queue, the data structure body corresponding to the fast table is cached to the fast table buffer queue, and the data structure body corresponding to the slow table is cached to the slow table buffer queue, as shown in the figure. Figure 2 As shown in the figure, there are 3 fast table buffer queues and 1 slow table buffer queue, wherein the data synchronization speed of the fast table is greater than that of the slow table. In this way, the influence of the slow warehousing table on the overall synchronization performance is reduced, the incremental synchronization throughput is improved, and the delay is reduced.

[0074] Further, the real-time synchronization speeds of each materialized view log table are further included to calculate an average real-time synchronization speed of all the materialized view log tables, and the fast table and slow table types of each materialized view log table are updated in real time based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all the materialized view log tables, specifically, the materialized view log table with a real-time synchronization speed greater than the average real-time synchronization speed of all the materialized view log tables is defined as a fast table, and the materialized view log table with a real-time synchronization speed less than the average real-time synchronization speed of all the materialized view log tables is defined as a slow table. In this way, the data structure bodies of each table can be adjusted to be cached into the fast table buffer queue or the slow table buffer queue according to the real-time synchronization speed of each materialized view log table, further reducing the influence of the slow table on the overall synchronization performance, improving the incremental synchronization throughput, and reducing the delay.

[0075] More specifically:

[0076] In the caching stage:

[0077] Since the materialized view log table is defined as a fast table or a slow table which can be changed according to the real-time synchronization speed, the table defined as a slow table at the last moment can be defined as a fast table at the next moment, and the synchronization speed is obtained in the warehousing stage, while the extraction and caching are continuously performed. Therefore, in the extraction and caching stage, the table corresponding to the extracted data structure body can be currently defined as a fast table, but the slow table buffer queue already has data corresponding to the table. If the data structure body is cached into the fast table buffer queue at this time, the data in the same materialized view log table can be simultaneously warehoused in the two buffer queues, which can cause the final synchronization data to be disordered. Therefore:

[0078] If the table corresponding to the extracted data structure body is currently defined as a fast table, and the slow table buffer queue does not have data corresponding to the table, the data structure body is cached into the fast table buffer queue.

[0079] If the table corresponding to the extracted data structure body is currently defined as a fast table, and the slow table buffer queue has data corresponding to the table, the extraction of the data in the table is suspended, and the extraction is continued after the data in the table in the slow table buffer queue is completely warehoused, and the table corresponding to the data structure body is re-judged as being currently defined as a fast table or a slow table during the extraction.

[0080] Similarly, since the materialized view log table is defined as a fast table or a slow table which can be changed according to the real-time synchronization speed, the table defined as a fast table at the last moment can be defined as a slow table at the next moment, and if the fast table buffer queue has data corresponding to the slow table, the warehousing speed of other fast tables can be affected. Therefore:

[0081] If the table corresponding to the extracted data structure body is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, the data structure body is cached to the slow table buffer queue.

[0082] If the table corresponding to the extracted data structure body is currently defined as a slow table, and there is data corresponding to the table in the fast table buffer queue, the extraction of data in the table is suspended, and the extraction is continued after the data in the fast table buffer queue is inserted into the slow table buffer queue, and the current definition of the table corresponding to the data structure body is re-judged as a fast table or a slow table.

[0083] In the warehousing stage:

[0084] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a slow table, the data structure body is inserted into the slow table buffer queue, and the warehousing of other data structure bodies in the fast table buffer queue is continued, so as to avoid the influence of the slow warehousing table on the overall synchronization performance.

[0085] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the data structure body is inserted into the slow table buffer queue, and the warehousing of other data structure bodies in the fast table buffer queue is continued, so as to avoid the synchronization data disorder caused by the warehousing of data of the same table in two buffer queues.

[0086] If the table corresponding to the data structure body in the fast table buffer queue to be currently warehoused is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the warehousing of the data structure body in the fast table buffer queue is continued.

[0087] The warehousing thread continues to warehouse the data structure bodies in the slow table buffer queue.

[0088] It can be seen that the application proposes the strategy of preferential extraction of "hot table" and preferential warehousing of "fast table": the application uses self-defined parameters and real-time running data to adjust the behavior of extraction and warehousing in real time. By using the weight mechanism, hot table data with high real-time requirements and frequent operations are preferentially extracted; by using real-time running data, the tables are distinguished between fast and slow, and the parallel warehousing strategy is adopted for different tables, which reduces the influence of the slow warehousing table on the overall synchronization performance, improves the throughput of incremental synchronization, and reduces the delay.

[0089] Embodiment two:

[0090] Referring to Figure 3 The embodiment provides a parallel incremental synchronization system, which comprises a log table establishing module, an extraction module, a buffer module, a warehousing module and an incremental data confirmation module connected in sequence, and the incremental data confirmation module is further connected with the log table establishing module.

[0091] a log table establishing module, configured to establish a materialized view log table for each data table to be synchronized;

[0092] an extraction module, configured to establish a plurality of extraction threads, and extract data in the plurality of materialized view log tables in parallel and in batches through the plurality of extraction threads, and store the data extracted in parallel and in batches in a plurality of data structures respectively, each of the data structures recording a start sequence number and an end sequence number of corresponding batch data;

[0093] a buffering module, configured to buffer the plurality of data structures in a plurality of buffering queues respectively;

[0094] a warehousing module, configured to establish a plurality of warehousing threads, and perform warehousing on the data structures buffered in the plurality of buffering queues in parallel through the plurality of warehousing threads;

[0095] an incremental data confirmation module, configured to confirm the incremental data to be synchronized and delete corresponding records in the materialized view log tables according to the start sequence number and the end sequence number recorded by each data structure after the data structure is warehoused.

[0096] Specifically,

[0097] The system further comprises a strategy module connected with the log table establishing module and the extraction module respectively;

[0098] The strategy module is configured to assign an initial weight corresponding to each materialized view log table, and divide each materialized view log table into a hot table and a normal table based on the initial weight corresponding to each materialized view log table.

[0099] The plurality of extraction threads established by the extraction module comprise hot table data extraction threads and normal table data extraction threads in a preset proportion, wherein the proportion of the hot table data extraction threads is greater than that of the normal table data extraction threads, the hot table data extraction threads preferentially extract data with a greater weight in all hot tables, and the normal table data extraction threads preferentially extract data with a greater weight in all normal tables.

[0100] The strategy module is further configured to statistically determine the amount of remaining synchronization data in each materialized view log table, and adjust the weight corresponding to each materialized view log table according to the amount of remaining synchronization data in each materialized view log table.

[0101] The strategy module is further connected with the buffering module;

[0102] The strategy module is further configured to initialize and define each materialized view log table as a fast table or a slow table, wherein the data synchronization speed of the fast table is greater than that of the slow table.

[0103] The plurality of buffer queues in the buffer module include a fast table buffer queue and a slow table buffer queue, and the data structure corresponding to the fast table is cached to the fast table buffer queue, and the data structure corresponding to the slow table is cached to the slow table buffer queue.

[0104] The strategy module is further configured to calculate the average real-time synchronization speed of all the materialized view log tables based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all the materialized view log tables, and update the fast table and the slow table of each materialized view log table in real time based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all the materialized view log tables.

[0105] The strategy module defines the materialized view log table with a real-time synchronization speed greater than the average real-time synchronization speed of all the materialized view log tables as a fast table, and defines the materialized view log table with a real-time synchronization speed less than the average real-time synchronization speed of all the materialized view log tables as a slow table.

[0106] The buffer module caches the data structure in the cache stage.

[0107] If the table corresponding to the extracted data structure is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the data structure is cached to the fast table buffer queue.

[0108] If the table corresponding to the extracted data structure is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the extraction of the data in the table is paused, and the extraction is continued after the data of the table in the slow table buffer queue is all warehoused.

[0109] If the table corresponding to the extracted data structure is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, the data structure is cached to the slow table buffer queue.

[0110] If the table corresponding to the extracted data structure is currently defined as a slow table, and there is data corresponding to the table in the fast table buffer queue, the extraction of the data in the table is paused, and the extraction is continued after the data of the table in the fast table buffer queue is all inserted into the slow table buffer queue.

[0111] The strategy module is further connected with the warehousing module, and the warehousing module is in the warehousing stage.

[0112] If the table corresponding to the data structure in the fast table buffer queue to be warehoused is currently defined as a slow table, the data structure is inserted into the slow table buffer queue, and the warehousing of other data structures in the fast table buffer queue is continued.

[0113] If the table corresponding to the data structure body in the fast table buffer queue currently to be stored is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, the data structure body is inserted into the slow table buffer queue, and the storage of other data structure bodies in the fast table buffer queue is continued;

[0114] If the table corresponding to the data structure body in the fast table buffer queue currently to be stored is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, the storage of the data structure body in the fast table buffer queue is continued.

[0115] The storage thread continues to store the data structure body in the slow table buffer queue.

[0116] It should be noted that the parallel incremental synchronization system provided in this embodiment is similar to the first embodiment, and will not be described in detail here.

[0117] Embodiment three:

[0118] Referring to Figure 4 The present embodiment also provides a storage medium having a computer program 601 stored thereon, and the instructions are executed by a processor to implement the steps of the parallel incremental synchronization method in the above embodiments. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The program can include the processes of the first embodiment when executed.

[0119] The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0120] Embodiment four:

[0121] Referring to Figure 5 The present embodiment also provides an electronic device, which can include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 can be connected by a bus or other means, Figure 4 for example, by a bus connection.

[0122] The processor 51 can be a central processing unit (CPU). The processor 51 can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or a combination thereof.

[0123] The memory 52 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiments of the present application. The processor 51 performs various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, implements the parallel incremental synchronization method in the above-mentioned embodiment one.

[0124] The memory 52 can include a program storage area and a data storage area, wherein the program storage area can store application programs required by the operation device and at least one function; and the data storage area can store data created by the processor 51 and the like. In addition, the memory 52 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 52 can optionally include a memory disposed remotely with respect to the processor 51, and these remote memories can be connected to the processor 51 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0125] The one or more modules are stored in the memory 52, and when executed by the processor 51, implement the parallel incremental synchronization method in the above-mentioned embodiment one.

[0126] The above-mentioned electronic device specific details can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiment one, which will not be repeated here.

[0127] The above-mentioned embodiments only describe the preferred embodiments of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements of the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application.

Claims

1. A parallel incremental synchronization method, characterized in that, Including the following steps: Create a materialized view log table for each data table to be synchronized; Multiple extraction threads are established, and data from multiple materialized view log tables are extracted in parallel and in batches through multiple extraction threads. The data extracted in parallel and in batches is stored in multiple data structures, and each data structure records the start sequence number and end sequence number of the corresponding batch of data. Cache multiple data structures into multiple buffer queues respectively; Multiple inbound threads are established, and the data structures cached in multiple buffer queues are inbound in parallel through multiple inbound threads. After inbound, the incremental data is confirmed according to the start sequence number and end sequence number recorded in each data structure, and the corresponding record in the materialized view log table is deleted. After creating a materialized view log table for each data table to be synchronized, assign an initial weight to each materialized view log table. Based on the initial weight, each materialized view log table is divided into hot tables and ordinary tables. Multiple extraction threads include a preset proportion of hot table data extraction threads and ordinary table data extraction threads, with the proportion of hot table data extraction threads being greater than that of ordinary table data extraction threads. Hot table data extraction threads prioritize extracting data from all hot tables with larger weights, and ordinary table data extraction threads prioritize extracting data from all ordinary tables with larger weights. It also includes real-time statistics of the remaining synchronized data in each materialized view log table, and adjusting the corresponding weights of each materialized view log table based on the remaining synchronized data in each materialized view log table. After creating a materialized view log table for each data table to be synchronized, each materialized view log table is initialized and defined as a fast table or a slow table. The multiple buffer queues include a fast table buffer queue and a slow table buffer queue. The data structure corresponding to the fast table is cached in the fast table buffer queue, and the data structure corresponding to the slow table is cached in the slow table buffer queue. The data synchronization speed of the fast table is greater than that of the slow table.

2. The parallel incremental synchronization method according to claim 1, characterized in that, It also includes real-time statistics of the real-time synchronization speed of each materialized view log table, in order to calculate the average real-time synchronization speed of all materialized view log tables, and to update the fast table and slow table types of each materialized view log table in real time based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all materialized view log tables.

3. The parallel incremental synchronization method according to claim 2, characterized in that, Materialized view log tables whose real-time synchronization speed is greater than the average real-time synchronization speed of all materialized view log tables are defined as fast tables, and materialized view log tables whose real-time synchronization speed is less than the average real-time synchronization speed of all materialized view log tables are defined as slow tables.

4. The parallel incremental synchronization method according to claim 3, characterized in that, During the caching stage: If the table corresponding to the extracted data structure is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, then the data structure is cached in the fast table buffer queue. If the table corresponding to the extracted data structure is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, then the extraction of data in the table will be paused, and the extraction will continue after all the data in the slow table buffer queue has been entered into the database. If the table corresponding to the extracted data structure is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, then the data structure is cached in the slow table buffer queue. If the table corresponding to the extracted data structure is currently defined as a slow table, and there is data corresponding to that table in the fast table buffer queue, then the extraction of data in that table will be paused. Extraction will continue after all the data in the fast table buffer queue of that table has been inserted into the slow table buffer queue.

5. The parallel incremental synchronization method according to claim 4, characterized in that, During the warehousing stage: If the table corresponding to the data structure in the fast table buffer queue that is currently to be inserted is defined as a slow table, then insert the data structure into the slow table buffer queue and continue to insert other data structures in the fast table buffer queue. If the table corresponding to the data structure to be inserted into the database is currently defined as a fast table and there is data corresponding to that table in the slow table buffer queue, then the data structure is inserted into the slow table buffer queue, and the insertion of other data structures in the fast table buffer queue continues. If the table corresponding to the data structure in the fast table buffer queue that is to be inserted is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, then the insertion of the data structure in the fast table buffer queue will continue. The database insertion thread pool continuously inserts data structures from the slow table buffer queue into the database.

6. A parallel incremental synchronization system, characterized in that, It includes a log table creation module, an extraction module, a buffer module, an ingestion module, and an incremental data confirmation module, which are connected in sequence. The incremental data confirmation module is also connected to the log table creation module. The log table creation module is used to create a materialized view log table for each data table to be synchronized; The extraction module is used to create multiple extraction threads and extract data from multiple materialized view log tables in parallel and in batches. The data extracted in parallel and in batches is stored in multiple data structures, and each data structure records the start sequence number and end sequence number of the corresponding batch of data. The buffer module is used to cache multiple data structures into multiple buffer queues respectively; The data entry module is used to create multiple data entry threads and use these threads to enter data structures cached in multiple buffer queues in parallel. The incremental data confirmation module is used to confirm the synchronized incremental data based on the start and end sequence numbers of each data structure record after the data structure is entered into the database, and to delete the corresponding records in the materialized view log table. It also includes a strategy module that connects to the log table creation module and the extraction module, respectively; The strategy module is used to assign initial weights to each materialized view log table and to divide each materialized view log table into hot tables and ordinary tables based on the initial weights. The extraction module establishes multiple extraction threads, including a preset ratio of hot table data extraction threads and ordinary table data extraction threads. The ratio of hot table data extraction threads is greater than that of ordinary table data extraction threads. The hot table data extraction threads prioritize extracting data from all hot tables with larger weights, and the ordinary table data extraction threads prioritize extracting data from all ordinary tables with larger weights. The strategy module is also used to count the amount of remaining synchronized data in each materialized view log table in real time, and adjust the weight of each materialized view log table according to the amount of remaining synchronized data in each materialized view log table. The strategy module is also connected to the buffer module; The strategy module is also used to initialize each materialized view log table as a fast table or a slow table, with the data synchronization speed of the fast table being greater than that of the slow table. The buffer module includes multiple buffer queues, including a fast table buffer queue and a slow table buffer queue. The data structures corresponding to the fast table are cached in the fast table buffer queue, and the data structures corresponding to the slow table are cached in the slow table buffer queue.

7. A parallel incremental synchronization system according to claim 6, characterized in that, The strategy module is also used to statistically analyze the real-time synchronization speed of each materialized view log table, calculate the average real-time synchronization speed of all materialized view log tables, and update the fast table and slow table types of each materialized view log table in real time based on the real-time synchronization speed of each materialized view log table and the average real-time synchronization speed of all materialized view log tables.

8. A parallel incremental synchronization system according to claim 7, characterized in that, In the strategy module, materialized view log tables with a real-time synchronization speed greater than the average real-time synchronization speed of all materialized view log tables are defined as fast tables, and materialized view log tables with a real-time synchronization speed less than the average real-time synchronization speed of all materialized view log tables are defined as slow tables.

9. A parallel incremental synchronization system according to claim 8, characterized in that, The buffer module in the caching phase: If the table corresponding to the extracted data structure is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, then the data structure is cached in the fast table buffer queue. If the table corresponding to the extracted data structure is currently defined as a fast table, and there is data corresponding to the table in the slow table buffer queue, then the extraction of data in the table will be paused, and the extraction will continue after all the data in the slow table buffer queue has been entered into the database. If the table corresponding to the extracted data structure is currently defined as a slow table, and there is no data corresponding to the table in the fast table buffer queue, then the data structure is cached in the slow table buffer queue. If the table corresponding to the extracted data structure is currently defined as a slow table, and there is data corresponding to that table in the fast table buffer queue, then the extraction of data in that table will be paused. Extraction will continue after all the data in the fast table buffer queue of that table has been inserted into the slow table buffer queue.

10. A parallel incremental synchronization system according to claim 9, characterized in that, The strategy module is also connected to the inbound module, which performs the following during the inbound phase: If the table corresponding to the data structure in the fast table buffer queue that is currently to be inserted is defined as a slow table, then insert the data structure into the slow table buffer queue and continue to insert other data structures in the fast table buffer queue. If the table corresponding to the data structure to be inserted into the database is currently defined as a fast table and there is data corresponding to that table in the slow table buffer queue, then the data structure is inserted into the slow table buffer queue, and the insertion of other data structures in the fast table buffer queue continues. If the table corresponding to the data structure in the fast table buffer queue that is to be inserted is currently defined as a fast table, and there is no data corresponding to the table in the slow table buffer queue, then the insertion of the data structure in the fast table buffer queue will continue. The data entry thread continuously enters data structures from the slow table buffer queue.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform a parallel incremental synchronization method as described in any one of claims 1-5.

12. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform a parallel incremental synchronization method as described in any one of claims 1-5.

Images