A system, method, device and storage medium for realizing incremental data synchronization

Abstract

This application discloses a system, method, device, and storage medium for incremental data synchronization. The system includes: at least one reading module and at least one writing module, with a corresponding relationship between the reading and writing modules; the reading module is used to obtain incremental data from a source database as data to be synchronized; determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each reading module constitutes the data to be synchronized; the writing module is used to obtain the data to be written from the corresponding reading module; and start multiple threads to control the writing of the data to be written to the target database based on the operation log of the data to be written. The writing module improves the speed of data writing to the target database by starting multiple threads to control data writing. The multi-process and multi-threaded data synchronization method can improve the speed of data synchronization, making the speed of data reading and data writing equal, and achieving timely data synchronization.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a system, method, device and storage medium for implementing incremental data synchronization. Background Art

[0002] When data is synchronized between databases, the data in the source database is read first, and then the data read from the source database is written to the target database to achieve data synchronization between databases.

[0003] In the prior art, the writing speed of the read data into the target database is slower than the reading speed of the data from the source database, resulting in unequal data reading and writing speeds, slow data synchronization between databases, and inability to synchronize data in a timely manner. Summary of the invention

[0004] In view of this, embodiments of the present application provide a system, method, device and storage medium for implementing incremental data synchronization, so as to achieve faster data synchronization between databases.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] A system for implementing incremental data synchronization, the system comprising: at least one reading module and at least one writing module, the reading module having a corresponding relationship with the writing module;

[0007] The reading module is used to obtain incremental data from the source database as data to be synchronized; determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each of the reading modules constitute the data to be synchronized;

[0008] The writing module is used to obtain the data to be written from the corresponding reading module; start multiple threads, and control the writing of the data to be written into the target database by using the multiple threads according to the operation log of the data to be written.

[0009] In a possible implementation, the system further includes a control module;

[0010] The control module is used to assign a serial number to each of the reading modules;

[0011] The reading module is specifically used to: perform a hash operation on the namespace field of the first target data in the data to be synchronized, and then perform a modulus operation on the number of reading modules to obtain a target value;

[0012] If the target value is the same as the sequence number of the current read module, the first target data is determined as the to-be-written data corresponding to the current read module.

[0013] In a possible implementation, the writing module is specifically used for:

[0014] Starting at least one data manipulation language thread and one data definition language thread;

[0015] Reading a write type in an operation log of second target data in the data to be written;

[0016] When the write type in the operation log of the second target data is a data manipulation language write type and the state of the data definition language thread is a non-write state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0017] When the write type in the operation log of the second target data is a data definition language write type, and when the states of each of the data manipulation language threads are in a non-write state, the data definition language thread is used to control the writing of the second target data into the target database according to the operation log of the second target data.

[0018] In a possible implementation, the writing module is specifically used to: when the writing type in the operation log of the second target data is a data manipulation language writing type, determine the state of the data definition language thread;

[0019] If the state of the data definition language thread is a non-writing state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0020] If the state of the data definition language thread is a writing state, after waiting for the state of the data definition language thread to change to a non-writing state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0021] and / or,

[0022] When the write type in the operation log of the second target data is a data definition language write type, determining the state of each of the data manipulation language threads;

[0023] If the states of the respective data manipulation language threads are all in a non-writing state, according to the operation log of the second target data, the data definition language thread is used to control the second target data to be written into the target database;

[0024] If the state of any of the data manipulation language threads is a writing state, wait until the state of the data manipulation language thread in the writing state changes to a non-writing state, and then, based on the operation log of the second target data, use the data definition language thread to control the writing of the second target data into the target database.

[0025] In a possible implementation, the writing module is further used to: determine a current writing progress in a process of controlling the writing of the to-be-written data into the target database by using the multiple threads;

[0026] The control module is also used to obtain the current writing progress of each of the writing modules.

[0027] In a possible implementation, the control module is further configured to:

[0028] When it is determined that the read module fails, obtaining the current writing progress of the writing module corresponding to the failed read module as the first writing progress;

[0029] Sending the first writing progress to the failed reading module, triggering the failed reading module to continue to determine the to-be-written data corresponding to the current reading module according to the first writing progress;

[0030] When it is determined that the writing module fails, obtaining the current writing progress of the failed writing module as the second writing progress;

[0031] The second writing progress is sent to the writing module where the fault occurs, triggering the writing module where the fault occurs to continue to control the writing of the to-be-written data into the target database by using the multiple threads according to the second writing progress.

[0032] A method for realizing incremental data synchronization, the method being applied to a reading module, the method comprising:

[0033] Obtain incremental data from the source database as data to be synchronized;

[0034] Determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each of the reading modules constitute the data to be synchronized;

[0035] The data to be written is sent to a corresponding writing module, so that the writing module starts multiple threads, and according to the operation log of the data to be written, the multiple threads are used to control the data to be written to be written into a target database.

[0036] A method for implementing incremental data synchronization, the method being applied to a writing module, the method comprising:

[0037] Acquire the data to be written from the corresponding reading module; the data to be written is the incremental data acquired by the reading module from the source database as the data to be synchronized, and is determined from the data to be synchronized, and the data to be written corresponding to each of the reading modules constitutes the data to be synchronized;

[0038] A plurality of threads are started, and according to the operation log of the data to be written, the plurality of threads are used to control the data to be written to be written into the target database.

[0039] A device for implementing incremental data synchronization comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned methods for implementing incremental data synchronization is implemented.

[0040] A computer-readable storage medium stores instructions, which, when executed on a terminal device, enable the terminal device to execute any one of the above-mentioned methods for realizing incremental data synchronization.

[0041] It can be seen that the embodiments of the present application have the following beneficial effects:

[0042] The system for realizing incremental data synchronization provided by the embodiment of the present application includes at least one reading module and a writing module corresponding to the reading module; in the system for realizing incremental data synchronization provided by the embodiment of the present application, the reading module is used to obtain incremental data from the source database as data to be synchronized, and then determine the data to be written corresponding to the current reading module; the writing module is used to obtain the data to be written from the corresponding reading module; finally, multiple threads are started, and according to the operation log of the data to be written, multiple threads are used to control the writing of the data to be written into the target database. By establishing at least one reading module and a writing module corresponding to the reading module in the incremental data synchronization system, a multi-process synchronization method can be adopted to realize that multiple reading modules can read data at the same time, and corresponding multiple writing modules can write data at the same time, thereby improving the reading speed and writing speed of data. After the writing module obtains the data to be written in the corresponding reading module, multiple threads are started to control the writing of data, and each writing module realizes the writing of data through multiple threads, further improving the speed of writing data into the target database. The data synchronization method of multiple processes and multiple threads can improve the speed of data synchronization, so that the speed of data reading and data writing is equal, and timely data synchronization is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a scenario for implementing incremental data synchronization provided in an embodiment of the present application;

[0044] Figure 2A schematic diagram of the structure of a system for implementing incremental data synchronization provided in an embodiment of the present application;

[0045] Figure 3 A flowchart of a method for a writing module to write data to be written into a target database provided in an embodiment of the present application;

[0046] Figure 4 A flowchart of a method for implementing incremental data synchronization provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of another method for implementing incremental data synchronization provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0049] After studying the traditional incremental data synchronization method, the inventor found that in the traditional incremental data synchronization method, the reading speed and writing speed of data are not equal, and the reading speed of data from the source database is greater than the writing speed of data into the target database. As a result, the data read from the source database cannot be written into the target database in a timely manner, and the speed of data synchronization between databases is slow, and timely data synchronization cannot be achieved. For example, when implementing incremental synchronization of a distributed document storage database, the synchronization of incremental data is achieved by pulling data from a circular data queue that stores incremental data. However, when performing data synchronization, the speed of writing into the target database is much lower than the speed of pulling data, resulting in the incremental data reading speed being greater than the data writing speed, and fast data synchronization cannot be achieved.

[0050] Based on this, an embodiment of the present application provides a system for realizing incremental data synchronization, wherein the system includes at least one reading module and at least one writing module, and the reading module corresponds to the writing module. The system for realizing incremental data synchronization provided by an embodiment of the present application includes: a reading module and a writing module. The reading module is used to obtain incremental data from a source database as data to be synchronized, and to determine the data to be written corresponding to the current reading module from the data to be synchronized; wherein the data to be synchronized is composed of the data to be written corresponding to each reading module; the writing module is used to obtain the data to be written from the corresponding reading module, and then start multiple threads, and according to the operation log of the data to be written, use multiple threads to control the writing of the data to be written into the target database.

[0051] In order to facilitate understanding of the method for implementing incremental data synchronization provided by the embodiment of the present application, Figure 1The method for implementing incremental data synchronization provided in the embodiment of the present application can be applied to the system 101 for implementing incremental data synchronization.

[0052] like Figure 1 As shown, this figure is a schematic diagram of the incremental data synchronization scenario provided by an embodiment of the present application.

[0053] The system 101 applied to incremental data synchronization includes at least one read module 102 and a write module 103 corresponding to the read module 102 .

[0054] In actual applications, the reading module 102 obtains incremental data from the source database 104 as data to be synchronized; determines the data to be written corresponding to the current reading module from the data to be synchronized. The writing module 103 obtains the data to be written from the corresponding reading module 102; starts multiple threads, and uses multiple threads to control the writing of the data to be written into the target database 105 according to the operation log of the data to be written.

[0055] It should be noted that the database types of the source database 103 and the target database 104 are not limited in the embodiment of the present application, and can be existing or future databases for storing data. Both the source database 103 and the target database 104 can be databases constructed by a server or a server cluster. The system 101 for implementing incremental data synchronization can be set in the same server. For example, the server where the source database 103 is located, or the server where the target database 104 is located, or a separately set server other than the server of the source database 103 and the server of the target database 104. The system 101 for implementing incremental data synchronization can also be set in a server cluster, for example, the read module 102 and the write module 103 can be respectively set in different servers of the server cluster, and different read modules 102 can be set in the same server or different servers, and different write modules 103 can also be set in the same server or different servers. The specific manner in which the system 101 for implementing incremental data synchronization is set in a server or a server cluster is not limited in the embodiment of the present application, and can be set according to the actual application scenario.

[0056] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0057] To facilitate understanding of the present application, a system for implementing incremental data synchronization provided by an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0058] First of all, it should be explained that an embodiment of the present application provides a system for implementing incremental data synchronization, the system includes at least one reading module and at least one writing module, the reading module and the writing module have a corresponding relationship, and the corresponding relationship can be a one-to-one corresponding relationship.

[0059] The read module is a module used to read data from the source database, and each read module has a corresponding write module, which is used to write the data to be synchronized by the corresponding read module into the target database.

[0060] The number of the read module is at least one. The number of the write module can be one or at least one. Through the at least one read module and the corresponding write module, data is transmitted between the read module and the corresponding write module, thereby realizing fast multi-process data synchronization.

[0061] See also Figure 2 As shown, the figure is a structural diagram of a system for implementing incremental data synchronization provided by an embodiment of the present application, and the system includes a reading module 201 and a writing module 202.

[0062] The reading module 201 is used to obtain incremental data from the source database as data to be synchronized; determine the data to be written corresponding to the current reading module from the data to be synchronized; and the data to be written corresponding to each reading module constitute the data to be synchronized.

[0063] When performing data synchronization, the incremental data in the source database needs to be synchronized to the target database as data, so that the target database has the same data as the source database.

[0064] The database with the incremental data to be synchronized is the source database, and the incremental data to be synchronized is the data to be synchronized. In the embodiment of the present application, the data to be synchronized can be the incremental data of MongoDB, which is a database based on distributed file storage, and a circular queue can be used to store incremental data.

[0065] In order to distinguish the data to be synchronized from other data in the source database, the data to be synchronized may be marked or the data range of the data to be synchronized may be limited. For example, all incremental data generated after the last data synchronization may be used as the data to be synchronized.

[0066] The reading module obtains the data to be synchronized from the source database. In a possible implementation, the reading module can obtain the data to be synchronized by triggering a synchronization button.

[0067] The reading module determines the to-be-written data corresponding to the current reading module from the to-be-synchronized data, and the to-be-written data corresponding to each reading module constitute the to-be-synchronized data.

[0068] In order to enable multiple read modules to read the data to be synchronized at the same time so that the corresponding write modules can write data at the same time, the data to be synchronized can be divided to obtain the data to be written corresponding to each read module, so that the corresponding data to be written can be read by the read module. The data to be written corresponding to each read module constitutes the data to be synchronized, so that it can be ensured that all the data to be synchronized can achieve data synchronization. In addition, in a possible situation, in order to prevent the data to be synchronized from being written repeatedly, there is no repeated data between the data to be written.

[0069] The reading module can determine the corresponding data to be written from the data to be synchronized, and then read the determined data to be written. The present application embodiment also provides an implementation method for the reading module to determine the data to be written corresponding to the current reading module from the data to be synchronized. For specific implementation methods, please refer to the following description.

[0070] By determining and reading the corresponding data to be written by the reading module, multiple reading modules can read their corresponding data to be written at the same time, thereby improving the reading speed of the synchronized data.

[0071] The writing module 202 is used to obtain the data to be written from the corresponding reading module; start multiple threads, and use the multiple threads to control the writing of the data to be written into the target database according to the operation log of the data to be written.

[0072] The writing module corresponds to the reading module and is a module used to write the data to be written into the target database. In a possible implementation, in order to prevent data synchronization from causing transmission confusion, the writing module and the reading module may correspond one to one.

[0073] After the reading module determines the corresponding data to be written, the writing module corresponding to the reading module obtains the data to be written through the reading module, so as to write the data to be written into the target database.

[0074] The writing module obtains the data to be written through the corresponding reading module, so as to realize the data transmission of the data to be written. On the basis of improving the reading speed of the data to be synchronized, the writing speed of the data to be synchronized is correspondingly improved.

[0075] The writing module starts multiple threads and uses multiple threads to control the writing of the data to be written into the target database according to the operation log of the data to be written.

[0076] It is understandable that the data reading speed may be slower than the data writing speed. In order to further improve the data writing speed, the writing module may start multiple threads to write the data to be written.

[0077] In the process of writing the data to be written into the target database, the writing module can use multiple threads to control the writing of the data to be written according to the operation log of the data to be written. There are multiple types of threads used for data synchronization adopted by the writing module, and some types of threads can use multiple threads of the same type to write the data to be written. The writing module can determine the thread to be used according to the operation log of the data to be written, and use multiple threads to control the writing of the data to be written into the target database.

[0078] In a possible implementation, the threads started by the writing module may include a DML (Data Manipulation Language) thread and a DDL (Data Definition Language) thread. Correspondingly, the embodiment of the present application also provides an implementation method in which the writing module uses threads to control the writing of the data to be written into the target database according to the operation log of the data to be written. Please refer to the following description for the specific implementation method.

[0079] Based on the relevant contents of the above-mentioned read module 201 and write module 202, it can be known that in the system for realizing incremental data synchronization provided by the embodiment of the present application, the read module obtains incremental data from the source database as the data to be synchronized, and then determines the data to be written corresponding to the current read module from the data to be synchronized, and the data to be written corresponding to each read module constitutes the data to be synchronized. By dividing the data to be synchronized, the read module is responsible for transmitting the corresponding data to be written to the write module corresponding to the read module, and the multi-process reading and writing of the data to be synchronized can be realized. The write module controls the writing of the data to be written to the target database through multiple threads, and the data to be written can be written into the target database by a multi-threaded writing method, which improves the writing speed of the data to be written. By adopting a multi-process and multi-threaded data synchronization method, the writing speed of the data to be synchronized can be improved on the basis of synchronously improving the reading speed and writing speed of the data to be synchronized, and the gap between the reading speed and the writing speed is reduced, so that fast data synchronization can be achieved between databases.

[0080] In a possible implementation, the threads used to write the data to be written into the target database may include a dml thread and a ddl thread, and the writing module may determine and control the writing of the data to be written according to the type of thread and the state of the thread.

[0081] The write module 202 can be specifically used to start at least one data manipulation language thread and one data definition language thread; read the write type in the operation log of the second target data in the data to be written; when the write type in the operation log of the second target data is the data manipulation language write type, when the state of the data definition language thread is a non-write state, according to the operation log of the second target data, use at least one data manipulation language thread to control the second target data to be written into the target database; when the write type in the operation log of the second target data is the data definition language write type, when the states of each data manipulation language thread are all in a non-write state, according to the operation log of the second target data, use the data definition language thread to control the second target data to be written into the target database.

[0082] The following is an introduction to the working principle of the above writing module through the process. Figure 3 As shown, the figure is a flow chart of a method for a writing module to write data to be written into a target database provided by an embodiment of the present application, including S301-S304:

[0083] S301: Start at least one data manipulation language thread and one data definition language thread.

[0084] The write module starts the dml thread and the ddl thread for writing the data to be written. Among them, the dml thread can support multi-threaded writing, and multiple dml threads can be used at the same time to write the data to be written, which will not affect the final result of writing the data to be written into the target database. However, the ddl thread cannot support multi-threaded writing. If multiple ddl threads are used to write the data to be written, it will cause errors in writing the data to be written. Therefore, the write module can start at least one dml thread and one ddl thread for writing the data to be written.

[0085] The writing module starts at least one dml thread and one ddl thread to write the data to be written, and can adopt a multi-threaded writing method to achieve the writing of the data to be written.

[0086] S302: Read the write type in the operation log of the second target data in the data to be written.

[0087] The writing module reads each piece of data to be written, i.e., the second target data, and the write type in the operation log, and determines the type of thread to be selected according to the write type. The second target data may be a piece of data obtained from the data to be written. After determining the write type in the operation log of the second target data and determining the type of thread to be selected according to the write type, the next piece of data to be written is selected as the second target data, and the corresponding write type and thread type are determined. This process is repeated until each piece of data to be written is used as the second target data, and the write type in the operation log and the corresponding thread type are determined.

[0088] Corresponding to the dml thread and ddl thread started by the write module, the write type of the second target data in the operation log of the second target data can be a dml write type or a ddl write type. When the write type in the operation log of the second target data is the dml write type, the dml thread can be used to write to the target database; when the write type in the operation log of the second target data is the ddl write type, the ddl thread can be used to write to the target database.

[0089] S303: When the write type in the operation log of the second target data is a data manipulation language write type and the state of the data definition language thread is a non-write state, at least one data manipulation language thread is used to control the writing of the second target data into the target database according to the operation log of the second target data.

[0090] When the write type in the operation log of the second target data is the dml write type, the write module needs to use the dml thread to write the second target data.

[0091] In order to determine whether a thread is available, the thread state can be divided into an asynchronous state and a synchronous state according to whether the thread is performing data synchronization. The asynchronous state means that the thread is not currently executing to write the data to be written to the target database, and the synchronous state means that the thread is executing to write the data to be written to the target database.

[0092] It is understandable that for the same write module, only threads of the same type can be used at the same time. After determining that the DML thread needs to be used to write the second target data, it can be determined whether the DML thread is available by judging the state of the DDL thread. When the state of the DDL thread is a non-writing state, the DML thread can be used to write the data to be written.

[0093] The writing module uses at least one dml thread to control the second target data to be written into the target database according to the operation log of the data to be written. The operation log contains the storage records of the data to be written and the operation records of the database. According to the operation log, the relevant information required to write the data to be written can be obtained.

[0094] Furthermore, when the write type of the second target data is a dml write type, the number of the corresponding dml threads may be at least one. In order to increase the writing speed of the second target data, at least one dml thread may be used to control the writing of the second target data into the target database.

[0095] S304: When the write type in the operation log of the second target data is a data definition language write type and the states of all data manipulation language threads are in a non-write state, the second target data is written into the target database using the data definition language thread according to the operation log of the second target data.

[0096] When the write type in the operation log of the second target data is a ddl write type, the write module needs to use a ddl thread to write the second target data.

[0097] The write module determines the state of the DML thread. Since there is at least one DML thread, it is necessary to make sure that the states of all DML threads are in a non-write state before using the DML thread to write data.

[0098] When the states of all DML threads are in a non-writing state, the DDL thread is used to control the second target data to be written into the target database according to the operation log of the data to be written.

[0099] Furthermore, the number of DDL threads started by the writing module is one, and the writing module can use the started DDL thread to control the writing of the second target data.

[0100] In an embodiment of the present application, at least one DML thread and one DDL thread are started by a write module, and the thread type used is determined according to the write type in the operation log in the second target data. Under the condition that the thread of the corresponding type can be used, the second target log is written into the target database according to the operation log of the data to be written. The write module can write the data to be written into the target database in a multi-threaded synchronization manner, thereby improving the writing speed of the data to be written. In this way, parity between the reading speed and the writing speed of the data to be synchronized can be achieved, thereby improving the speed of data synchronization between databases.

[0101] In a possible implementation, the writing module 202 is specifically configured to determine the state of the data definition language thread when the writing type in the operation log of the second target data is a data manipulation language writing type;

[0102] If the state of the data definition language thread is a non-writing state, using at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data;

[0103] If the state of the data definition language thread is a writing state, after waiting for the state of the data definition language thread to change to a non-writing state, according to the operation log of the second target data, using at least one data manipulation language thread to control the second target data to be written into the target database;

[0104] and / or,

[0105] When the write type in the operation log of the second target data is a data definition language write type, determining the state of each data manipulation language thread;

[0106] If the states of the various data manipulation language threads are all in a non-writing state, the second target data is controlled to be written into the target database using the data definition language thread according to the operation log of the second target data;

[0107] If the state of any DML thread is writing state, wait until the state of the DML thread in writing state changes to non-writing state, then use the DDL thread to control the writing of the second target data into the target database according to the operation log of the second target data.

[0108] Specifically, the working principle of the writing module writing the second target data of the write type of dml write type into the target database is introduced in the form of steps, which specifically includes the following three steps:

[0109] A1: When the write type in the operation log of the second target data is the dml write type, determine the state of the ddl thread.

[0110] When the write module determines that the write type in the operation log of the second target data is the dml write type, it can determine whether the dml thread is available by determining the state of the ddl thread.

[0111] A2: If the state of the DDL thread is a non-writing state, at least one DML thread is used to control the second target data to be written into the target database according to the operation log of the data to be written.

[0112] If the state of the ddl thread is a non-writing state, the writing module can use the dml thread to write the data to be written. According to the operation log of the data to be written, at least one dml thread is used to control the second target data to be written into the target database.

[0113] A3: If the state of the DDL thread is the writing state, wait until the state of the DDL thread changes to the non-writing state, and then use at least one DML thread to control the second target data to be written into the target database according to the operation log of the data to be written.

[0114] If the state of the DDL thread is in the writing state, it means that the writing module is using the DDL thread to write data, and the DML thread cannot be used to write the second target data. It is necessary to wait for the state of the DDL thread to change to the non-writing state before using the DML thread to write the second target data. According to the operation log of the data to be written, at least one DML thread is used to control the writing of the second target data.

[0115] In addition, the working principle of the writing module writing the second target data of the write type of the ddl write type into the target database is introduced in the form of steps, which specifically includes the following three steps:

[0116] Specifically, the method of writing the second target data of the write type of the ddl write type into the target database may include the following three steps:

[0117] B1: When the write type in the operation log of the second target data is the DDL write type, determine the status of each DML thread.

[0118] When the write type in the operation log of the second target data is the DDL write type, whether the DDL thread can be used to write data is determined by determining the status of each DML thread. In addition, since the write module can use multiple DML threads to write the corresponding second target data of the DML write type, it is necessary to determine the status of each DML thread.

[0119] B2: If the status of each DML thread is a non-writing state, the DDL thread is used to control the second target data to be written into the target database according to the operation log of the data to be written.

[0120] If the states of all dml threads are in a non-writing state, the dml thread can be used to write the second target data. According to the operation log of the data to be written, the ddl thread is used to control the second target data to be written into the target database.

[0121] B3: If the state of any DML thread is in the writing state, wait for the state of the DML thread in the writing state to change to the non-writing state, and then use the DDL thread to control the second target data to be written into the target database according to the operation log of the data to be written.

[0122] When the state of any dml thread is in the write state, it indicates that the write module uses the dml thread to write data. It is necessary to wait for the state of the dml thread in the write state to change to the non-write state, that is, after the states of all dml threads are in the non-write state, according to the operation log of the data to be written, use the ddl thread to control the second target data to be written into the target database. In an embodiment of the present application, the state of other threads is judged by the control module, the state of the thread corresponding to the write type in the operation log is determined, and then it is determined whether the thread corresponding to the write type in the operation log is available. In this way, confusion in the use of threads can be prevented, and orderly use of multiple threads to write the second target data can be achieved. By determining the threads that can be used and then controlling the writing of the second target data, it is possible to control the threads corresponding to different write types, and the data to be written can be written into the target database in a multi-threaded synchronous manner, thereby improving the writing speed of the data to be written.

[0123] In a possible implementation, the system for implementing incremental data synchronization may further include a control module, and the control module may control the reading module and the writing module.

[0124] The reading module is specifically used to assign a serial number to each reading module;

[0125] The reading module is specifically used to: perform a hash operation on the namespace field of the first target data in the data to be synchronized, and then perform a modulo operation on the number of the reading modules to obtain a target value;

[0126] If the target value is the same as the serial number of the current read module, the first target data is determined as the data to be written corresponding to the current read module. Specifically, the control module can set data filtering conditions for each read module, that is, can set a corresponding serial number for each read module to achieve the correspondence between the read module and the data to be written.

[0127] For example, the control module assigns a natural number serial number starting from 0 to each reading module, so that each reading module has a corresponding serial number identification, namely: model_0, model_1, model_2, ... model_n-1, wherein n is the total number of reading modules.

[0128] The read module performs a hash operation on the namespace field of the first target data and then takes the modulus of n to obtain the target value. If the target value is the same as the serial number of the current read module, the target data is determined as the data to be written corresponding to the current read module. Wherein, n is the number of read modules, and the first target data is each data in the data to be synchronized. The first target data can be a piece of data obtained from the data to be synchronized. After being determined as the data to be written of the corresponding read module, the next piece of data to be synchronized is selected as the first target data, and the corresponding read module is determined, and so on, until each piece of data in the data to be synchronized is determined as the first target data and the corresponding read module is determined, and the determination of the data to be written is completed.

[0129] Correspondingly, each incremental data in the data to be synchronized has a field for identifying the ownership of the data, which can be called a namespace field. The namespace field is a string, and the namespace field can be converted into a numerical value by performing a hash operation on the namespace field. Then, the numerical value corresponding to the namespace field is used to perform a modulo operation on n to obtain the target value. Each piece of data in the data to be synchronized is used as the first target data, and the corresponding reading module is determined according to the target value of the first target data.

[0130] In order to achieve a relatively even data distribution, a hash calculation can be performed on the namespace field in the first target data, and the obtained value can be calculated modulo n to obtain the target value. The target value is then compared with the serial number of the current read module. If the target value is the same as the serial number of the current read module, the first target data is determined as the data to be written corresponding to the current read module, and the current read module transmits the corresponding data to be written to the corresponding write module.

[0131] Taking the serial number identification of the control module and the document identification of the data document as an example, the number of read modules is 10, that is, n is 10. If the namespace field of the first target data is hashed, the value corresponding to the namespace field of the first target data is 5, and then 5 is modulo 10, the target value is 5. This first target data is the data to be written corresponding to the read module with the serial number identification model_5. By analogy, each piece of data to be synchronized has a corresponding read module.

[0132] In an embodiment of the present application, a data filtering condition is set for each read module through a control module, a serial number is assigned to each read module in the read module, and further, according to the target value of the first target data, the data to be written corresponding to the read module is determined. By performing a hash operation on the namespace field of the first target data and then performing a modulo calculation on n, the corresponding read module can be determined according to the obtained target value. In this way, each piece of data to be synchronized has a corresponding read module, and the data to be synchronized with the same namespace field can correspond to the same read module. By determining the data to be written corresponding to the read module, it is convenient for multiple read modules to perform multi-process data reading and corresponding data writing, thereby improving the speed of data reading.

[0133] During the process of synchronizing the data to be synchronized in the source database to the target database, the progress of writing data can also be monitored.

[0134] The writing module is also used to: determine the current writing progress in the process of controlling the writing of the data to be written into the target database by using multiple threads;

[0135] The control module is also used to obtain the current writing progress of each writing module.

[0136] The current writing progress determined by the writing module may be the writing progress of the data to be written that has been written into the target database. Specifically, the writing module may use the timestamp in the operation log of the data to be written that has been written into the target database as the current writing progress. When new data to be written is successfully written into the target database, the current writing progress is updated using the timestamp in the operation log of the latest successfully written data to be written.

[0137] Each writing module determines the current writing progress of the corresponding data to be written, and sends the current writing progress to the control module. The control module can determine the current writing progress of each writing module by obtaining the current writing progress of each writing module, and adjust the data reading and writing according to the current writing progress of each writing module.

[0138] When synchronizing data between databases, the reading module or the writing module may fail, and corresponding adjustments need to be made to the reading and writing of data.

[0139] Furthermore, the control module is also used for:

[0140] When it is determined that the read module fails, the current write progress of the write module corresponding to the failed read module is obtained as the first write progress;

[0141] Sending the first writing progress to the failed reading module, triggering the failed reading module to continue to determine the to-be-written data corresponding to the current reading module according to the first writing progress;

[0142] When it is determined that the writing module fails, the current writing progress of the failed writing module is obtained as the second writing progress;

[0143] Send the second writing progress to the failed writing module, triggering the failed writing module to continue to use multiple threads to control the writing of the data to be written into the target database according to the second writing progress

[0144] In a possible implementation, when the control module determines that the reading module fails, the control module may perform the following two steps:

[0145] C1: When it is determined that a read module fails, the current write progress of the write module corresponding to the failed read module is obtained as the first write progress.

[0146] The control module can perform fault judgment based on the heartbeat signal sent by the reading module. If the heartbeat signal sent by the reading module is abnormal, it can be determined that the reading module has a fault.

[0147] When a read module fails, it may affect the corresponding write module from acquiring the data to be written. The control module may obtain the current writing progress of the write module corresponding to the failed read module, and use the current writing progress of the corresponding write module as the first writing data, so as to re-determine the data to be written to be read by the read module according to the current writing progress of the corresponding write module.

[0148] C2: Sending the first writing progress to the failed reading module, triggering the failed reading module to continue to determine the to-be-written data corresponding to the current reading module according to the first writing progress.

[0149] The control module sends the first writing progress to the reading module where the fault occurs. The reading module where the fault occurs can determine the data to be written that has been successfully written by the corresponding writing module according to the first writing progress, thereby determining the data to be written corresponding to the current reading module, so as to further write the data to be written that has not been successfully written, thereby restarting the reading module where the fault occurs.

[0150] In another possible implementation, when the control module determines that the writing module fails, the control module may perform the following two steps:

[0151] D1: When it is determined that a write module fails, the current write progress of the failed write module is obtained as the second write progress.

[0152] The control module can judge the fault of the write module according to the heartbeat information sent by the write module. If the control module detects that the heartbeat signal sent by the write module is abnormal, it is judged that the write module has a fault.

[0153] When a write module fails, the write module cannot normally write the data to be written, and the writing of the data to be written needs to be readjusted according to the current writing progress.

[0154] The control module obtains the current writing progress of the failed writing module as the second writing progress, and uses the second writing progress to adjust the failed writing module.

[0155] D2: Sending the second writing progress to the failed writing module, triggering the failed writing module to continue to control the writing of the to-be-written data into the target database using multiple threads according to the second writing progress.

[0156] The control module sends the second writing progress to the failed writing module to trigger the failed writing module to restart. The failed writing module continues to control the unwritten data to be written to the target database using multiple threads according to the second writing progress.

[0157] In an embodiment of the present application, the write module obtains the current write progress and sends it to the control module. The control module determines whether a fault occurs in the read module and the write module. If a fault occurs in the read module or the write module, the control module can restart the failed read module or the failed write module based on the current write progress of the write module corresponding to the failed read module, or the current write progress of the failed write module. The restart of the read module or the write module can be achieved on the basis of ensuring the continuity of the progress of data synchronization. It can not only increase the speed of data synchronization, but also ensure the correctness of data synchronization. Based on the relevant content of the above-mentioned system for realizing incremental data synchronization, an embodiment of the present application also provides a method for realizing incremental data synchronization, which is applied to the read module, and is described below in conjunction with the accompanying drawings. See. Figure 4 , which is a flow chart of a method for implementing incremental data synchronization provided by an embodiment of the present application, the method comprising S401-S403:

[0158] S401: Acquire incremental data from a source database as data to be synchronized.

[0159] S402: Determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each of the reading modules constitute the data to be synchronized.

[0160] S403: Send the data to be written to a corresponding writing module, so that the writing module starts multiple threads, and controls the data to be written to a target database by using the multiple threads according to the operation log of the data to be written.

[0161] Optionally, determining the data to be written corresponding to the current reading module from the data to be synchronized includes:

[0162] After performing a hash operation on the namespace field of the first target data in the data to be synchronized, taking the modulus of the number of read modules to obtain a target value;

[0163] If the target value is the same as the sequence number of the current read module, the first target data is determined as the to-be-written data corresponding to the current read module.

[0164] Optionally, the method further includes:

[0165] When the control module determines that the current read module fails, the control module obtains a first write progress from the control module; the first write progress is the current write progress obtained by the control module from the write module corresponding to the current read module;

[0166] Continue to determine the to-be-written data corresponding to the current reading module according to the first writing progress.

[0167] Based on the above-mentioned system for realizing incremental data synchronization, the present application embodiment also provides a method for realizing incremental data synchronization, which is applied to the writing module and is described below with reference to the accompanying drawings. Figure 5 , which is a flow chart of a method for implementing incremental data synchronization provided by an embodiment of the present application, the method comprising S501-S502:

[0168] S501: Acquire data to be written from a corresponding reading module; the data to be written is incremental data acquired by the reading module from a source database as data to be synchronized, and is determined from the data to be synchronized, and the data to be written corresponding to each of the reading modules constitutes the data to be synchronized;

[0169] S502: Start multiple threads, and control the data to be written to a target database by using the multiple threads according to the operation log of the data to be written.

[0170] Optionally, starting a plurality of threads and controlling the writing of the data to be written into a target database by using the plurality of threads according to the operation log of the data to be written includes:

[0171] Starting at least one data manipulation language thread and one data definition language thread;

[0172] Reading a write type in an operation log of second target data in the data to be written;

[0173] When the write type in the operation log of the second target data is a data manipulation language write type and the state of the data definition language thread is a non-write state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0174] When the write type in the operation log of the second target data is a data definition language write type, and when the states of each of the data manipulation language threads are in a non-write state, the data definition language thread is used to control the writing of the second target data into the target database according to the operation log of the second target data.

[0175] Optionally, when the write type in the operation log of the second target data is a data manipulation language write type and the state of the data definition language thread is a non-write state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the writing of the second target data into the target database includes:

[0176] When the write type in the operation log of the second target data is a data manipulation language write type, determining the state of the data definition language thread;

[0177] If the state of the data definition language thread is a non-writing state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0178] If the state of the data definition language thread is a writing state, after waiting for the state of the data definition language thread to change to a non-writing state, according to the operation log of the second target data, using the at least one data manipulation language thread to control the second target data to be written into the target database;

[0179] and / or,

[0180] When the write type in the operation log of the second target data is a data definition language write type, and the states of the data manipulation language threads are all in a non-write state, according to the operation log of the second target data, using the data definition language thread to control the second target data to be written into the target database, comprises:

[0181] When the write type in the operation log of the second target data is a data definition language write type, determining the state of each of the data manipulation language threads;

[0182] If the states of the respective data manipulation language threads are all in a non-writing state, according to the operation log of the second target data, the data definition language thread is used to control the second target data to be written into the target database;

[0183] If the state of any of the data manipulation language threads is a writing state, wait until the state of the data manipulation language thread in the writing state changes to a non-writing state, and then, based on the operation log of the second target data, use the data definition language thread to control the writing of the second target data into the target database.

[0184] Optionally, the method further includes:

[0185] In the process of controlling the writing of the data to be written into the target database by using the multiple threads, determining the current writing progress;

[0186] The current writing progress is sent to the control module.

[0187] Optionally, the method further includes:

[0188] When the control module determines that the current writing module fails, a second writing progress is obtained from the control module; the second writing progress is the current writing progress obtained by the control module from the current writing module;

[0189] The plurality of threads are continued to be used to control the writing of the data to be written into the target database according to the second writing progress.

[0190] In addition, an embodiment of the present application also provides a device for implementing incremental data synchronization, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for implementing incremental data synchronization as described above is implemented.

[0191] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the method for implementing incremental data synchronization as described above.

[0192] In the method for realizing incremental data synchronization provided by the embodiment of the present application, the reading module is used to obtain incremental data from the source database as the data to be synchronized, and then determine the data to be written corresponding to the current reading module; the writing module is used to obtain the data to be written from the corresponding reading module; finally, multiple threads are started, and according to the operation log of the data to be written, multiple threads are used to control the writing of the data to be written into the target database. By establishing at least one reading module and a writing module corresponding to the reading module in the incremental data synchronization system, a multi-process synchronization method can be adopted to realize that multiple reading modules can read data at the same time, and the corresponding multiple writing modules can write data at the same time, thereby improving the reading speed and writing speed of the data. After the writing module obtains the data to be written in the corresponding reading module, it starts multiple threads to control the writing of the data, and each writing module realizes multi-threaded data writing, further improving the speed of writing data into the target database. The speed of data synchronization can be improved by the multi-process and multi-threaded data synchronization method, so that the speed of data reading and data writing is equal, and timely data synchronization is realized.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0194] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0195] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0196] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0197] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system for realizing incremental data synchronization, characterized in that: The system comprises: at least one reading module and at least one writing module, wherein the reading module has a corresponding relationship with the writing module; The reading module is used to obtain incremental data from the source database as data to be synchronized; determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each of the reading modules constitute the data to be synchronized; The writing module is used to obtain the data to be written from the corresponding reading module; start at least one data manipulation language thread and one data definition language thread; read the writing type in the operation log of the second target data in the data to be written; When the write type in the operation log of the second target data is a data manipulation language write type, determining the state of the data definition language thread; if the state of the data definition language thread is a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the state of the data definition language thread is a write state, after waiting for the state of the data definition language thread to be changed to a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; When the write type in the operation log of the second target data is a data definition language write type, determine the status of each of the data manipulation language threads; if the status of each of the data manipulation language threads is a non-write status, use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the status of any of the data manipulation language threads is a write status, wait for the status of the data manipulation language thread in the write status to change to a non-write status, and then use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data.

2. The system according to claim 1, characterized in that The system also includes a control module; The control module is used to assign a serial number to each of the reading modules; The reading module is specifically used to: perform a hash operation on the namespace field of the first target data in the data to be synchronized, and then perform a modulus operation on the number of reading modules to obtain a target value; If the target value is the same as the sequence number of the current read module, the first target data is determined as the to-be-written data corresponding to the current read module.

3. The system according to claim 2, characterized in that The writing module is also used to: determine the current writing progress in the process of controlling the writing of the data to be written into the target database by using multiple threads; The control module is also used to obtain the current writing progress of each of the writing modules.

4. The system according to claim 3, characterized in that The control module is also used for: When it is determined that the read module fails, obtaining the current writing progress of the writing module corresponding to the failed read module as the first writing progress; Sending the first writing progress to the failed reading module, triggering the failed reading module to continue to determine the to-be-written data corresponding to the current reading module according to the first writing progress; When it is determined that the writing module fails, obtaining the current writing progress of the failed writing module as the second writing progress; The second writing progress is sent to the writing module where the fault occurs, triggering the writing module where the fault occurs to continue to control the writing of the to-be-written data into the target database by using the multiple threads according to the second writing progress.

5. A method for realizing incremental data synchronization, characterized in that: The method is applied to a reading module, and the method comprises: Obtain incremental data from the source database as data to be synchronized; Determine the data to be written corresponding to the current reading module from the data to be synchronized; the data to be written corresponding to each of the reading modules constitute the data to be synchronized; Sending the data to be written to a corresponding writing module, so that the writing module: starts at least one data manipulation language thread and one data definition language thread, and reads the writing type in the operation log of the second target data in the data to be written; When the write type in the operation log of the second target data is a data manipulation language write type, determining the state of the data definition language thread; if the state of the data definition language thread is a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the state of the data definition language thread is a write state, after waiting for the state of the data definition language thread to be changed to a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; When the write type in the operation log of the second target data is a data definition language write type, determine the status of each of the data manipulation language threads; if the status of each of the data manipulation language threads is a non-write status, use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the status of any of the data manipulation language threads is a write status, wait for the status of the data manipulation language thread in the write status to change to a non-write status, and then use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data.

6. A method for realizing incremental data synchronization, characterized in that: The method is applied to a writing module, and the method comprises: Acquire the data to be written from the corresponding reading module; the data to be written is the incremental data acquired by the reading module from the source database as the data to be synchronized, and is determined from the data to be synchronized, and the data to be written corresponding to each of the reading modules constitutes the data to be synchronized; Start at least one data manipulation language thread and one data definition language thread; read the write type in the operation log of the second target data in the data to be written; When the write type in the operation log of the second target data is a data manipulation language write type, determining the state of the data definition language thread; if the state of the data definition language thread is a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the state of the data definition language thread is a write state, after waiting for the state of the data definition language thread to be changed to a non-write state, using the at least one data manipulation language thread to control the second target data to be written into the target database according to the operation log of the second target data; When the write type in the operation log of the second target data is a data definition language write type, determine the status of each of the data manipulation language threads; if the status of each of the data manipulation language threads is a non-write status, use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data; if the status of any of the data manipulation language threads is a write status, wait for the status of the data manipulation language thread in the write status to change to a non-write status, and then use the data definition language thread to control the second target data to be written into the target database according to the operation log of the second target data.

7. A device for realizing incremental data synchronization, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for realizing incremental data synchronization as described in claim 5 or the method for realizing incremental data synchronization as described in claim 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on the terminal device, the terminal device executes the method for realizing incremental data synchronization as claimed in claim 5, or the method for realizing incremental data synchronization as claimed in claim 6.

Images