Data synchronization method and device, equipment and storage medium

Abstract

The invention provides a data synchronization method and device, equipment and a storage medium, and relates to the technical field of data storage. The method comprises the steps that N incremental operation record sets arranged according to the time sequence are obtained from an incremental log of a source database, N is an integer larger than 1, and the Nth incremental operation record set is a current to-be-synchronized incremental operation record set. The method further comprises the step of determining a target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set, and the target incremental operation record set is the incremental operation record set which is successfully synchronized to the target database last time. And synchronizing the incremental operation record set after the target incremental operation record set in the N incremental operation record sets to the target database. According to the embodiment of the invention, when the incremental operation record set is synchronized, the synchronization abnormity can be quickly found and automatically repaired, so that accurate synchronization between the databases is realized, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application mainly relate to the field of data storage. More specifically, the embodiments of the present application relate to a data synchronization method, apparatus, device and storage medium. Background Art

[0002] In the rapid development of various applications, in order to facilitate the management and use of data, users usually use databases to store these data. Databases are used to store data collections electronically. It can contain any type of data, including text, numbers, images, videos, and files. As user needs increase, the functions and performance of databases are also improving rapidly. The improvement of database functions and performance is very important for organizations and users who use databases, because appropriate functions can significantly improve users' ability to process data and save users' time and resources.

[0003] With the development of database technology, multiple storage systems are often used for complex business systems. For example, the first type of database is used as a business transaction database to carry the business, and the second type of database is used as a full-text search analysis database. However, there are still many technical problems that need to be solved when performing collaborative operations on data services between databases. For example, how to improve the data synchronization performance between databases is an important technical issue. Summary of the invention

[0004] An embodiment of the present application provides a data synchronization solution.

[0005] According to the first aspect of the present application, a data synchronization method is provided. The method is applied to a database system, which includes a source database and a target database. The method includes: obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database, N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently; based on the checksum of the incremental operation record set before the Nth incremental operation record set, determining the target incremental operation record set, the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently; and synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

[0006] In this way, when synchronizing the Nth incremental operation record set, it is ensured that the previous incremental operation record set and the N record sets are synchronized to the target database, so that when synchronizing the incremental operation record sets in real time, it is possible to quickly discover synchronization anomalies through consistency verification methods and automatically repair the data with synchronization anomalies, solve abnormal problems such as rollback of the database, ensure the accurate synchronization of operations and data between databases, and improve user experience.

[0007] In some embodiments, the method further includes: generating a check code for the Nth incremental operation record set; and after determining that the Nth incremental operation record set has been successfully synchronized to the target database, sending the check code for the Nth incremental operation record set to the target database, and the check code for the Nth incremental operation record set is used to be stored in the target database. In this way, the generation of the check code and the storage of the check code in the target database can be realized, so that it can be used to realize consistency verification, so as to quickly discover synchronization anomalies and automatically repair data with synchronization anomalies.

[0008] In some embodiments, based on the check code of the incremental operation record set before the Nth incremental operation record set, determining the target incremental operation record set includes: obtaining a comparison check code from the target database, the comparison check code is stored in the target database, and is the check code of the latest incremental operation record set that has been successfully synchronized to the target database; and determining the target incremental operation record set by searching for the same check code as the comparison check code in the check code of the incremental operation record set before the Nth incremental operation record set, the check code of the target incremental operation record set is the same as the comparison check code. In this way, the incremental operation record set that needs to be synchronized can be quickly and accurately determined by using the check code, so as to solve the synchronization anomaly and automatically repair the data with the synchronization anomaly.

[0009] In some embodiments, the source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in intermediate formats, and obtaining the N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: using a parser matching the type of the source database to obtain the Nth incremental operation record set in the source format from the incremental log of the source database; and converting the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format. In this way, synchronization of incremental operation records between multiple different types of databases can be achieved.

[0010] In some embodiments, synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: using an adapter matching the type of the target database, converting the incremental operation record set in the Nth intermediate format into the incremental operation record set in the Nth target format; and sending the incremental operation record set in the Nth target format to the target database, so that the target database executes the incremental operation corresponding to the incremental operation record set in the Nth target format. In this way, synchronization of incremental operation records between multiple different types of databases can be achieved.

[0011] In some embodiments, synchronizing the incremental operation record set after the target incremental operation record set among N incremental operation record sets to the target database includes: screening the incremental operation record in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format according to the identifier of the target data table; converting the incremental operation record in the intermediate format related to the target data table into the incremental operation record in the target format related to the target data table; and sending the incremental operation record in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record in the target format related to the target data table. In this way, the synchronization of incremental operation records between data tables in multiple different types of databases can be achieved.

[0012] In some embodiments, obtaining N sets of incremental operation records arranged in chronological order from the incremental log of the source database includes: obtaining N sets of incremental operation records arranged in chronological order from the incremental log of the source database according to the identifier of the target data table. In this way, synchronization of incremental operation records between data tables of databases of the same type can be achieved.

[0013] In some embodiments, synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: filtering the incremental operation record related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and sending the incremental operation record related to the target data table to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record related to the target data table. In this way, synchronization of incremental operation records between data tables between databases of the same type can be achieved.

[0014] In some embodiments, synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: sending the Nth incremental operation record set to the target database so that the target database performs the incremental operation corresponding to the Nth incremental operation record set. In this way, the synchronization of the Nth incremental operation record set between databases can be accurately achieved.

[0015] In some embodiments, generating a verification code of the Nth incremental operation record set includes: generating a verification code of the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set. In this way, the verification information of the incremental operation record set can be quickly and accurately obtained for verifying the incremental operation record set.

[0016] According to the second aspect of the present application, a data synchronization device is provided. The device is applied to a database system, the database system includes a source database and a target database, the data synchronization device is deployed on a cloud management platform, or a source database, or a target database, and the device includes: an incremental operation record set acquisition unit, configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently; a target incremental operation record set determination unit, configured to determine the target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set, the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently; and an incremental operation record set synchronization unit, configured to synchronize the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

[0017] In some embodiments, the data synchronization device also includes: a verification code generation unit, configured to generate a verification code for the Nth incremental operation record set; and a verification code sending unit, configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0018] In some embodiments, the target incremental operation record set determination unit includes: a comparison verification code acquisition unit, configured to obtain a comparison verification code from a target database, the comparison verification code is stored in the target database, and is the verification code of the latest incremental operation record set that has been successfully synchronized to the target database; and a verification code search unit, configured to determine the target incremental operation record set by searching for a verification code identical to the comparison verification code in the verification code of the incremental operation record set before the Nth incremental operation record set, and the verification code of the target incremental operation record set is the same as the comparison verification code.

[0019] In some embodiments, the source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in intermediate formats, and the incremental operation record set acquisition unit includes: a parser-based incremental operation record set acquisition unit, configured to use a parser that matches the type of the source database to acquire the Nth incremental operation record set in the source format from the incremental log of the source database; and a first incremental operation record set conversion unit, configured to convert the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format.

[0020] In some embodiments, the incremental operation record set synchronization unit includes: a second incremental operation record set conversion unit, configured to convert the incremental operation record set in the Nth intermediate format into the incremental operation record set in the Nth target format using an adapter that matches the type of the target database; and an incremental operation record set sending unit, configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

[0021] In some embodiments, the incremental operation record set synchronization unit includes: a first incremental operation record screening unit, configured to screen out incremental operation records in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table; and a first incremental operation record sending unit, configured to send the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record in the target format related to the target data table.

[0022] In some embodiments, the incremental operation record set acquisition unit includes: a data table-based incremental operation record set acquisition unit configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

[0023] In some embodiments, the incremental operation record set synchronization unit includes: a second incremental operation record screening unit, configured to screen out incremental operation records related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and a second incremental operation record sending unit, configured to send the incremental operation records related to the target data table to the target database, so as to enable the target database to perform incremental operations corresponding to the incremental operation records related to the target data table.

[0024] In some embodiments, the incremental operation record set synchronization unit includes: a target database synchronization unit configured to send the Nth incremental operation record set to the target database, so as to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

[0025] In some embodiments, the verification code generation unit includes: a verification code generation unit for an incremental operation record set, configured to generate a verification code for an Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

[0026] According to the third aspect of the present application, a computing device cluster is also provided, comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device being used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect of the present application.

[0027] According to a fourth aspect of the present application, a computer program product comprising instructions is also provided, wherein when the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect of the present application.

[0028] According to the fifth aspect of the present application, a computer-readable storage medium is also provided, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect of the present application.

[0029] It can be understood that the apparatus of the second aspect, the computing device cluster of the third aspect, the computer program product of the fourth aspect, or the computer-readable storage medium of the fifth aspect provided above is used to execute the method provided in the first aspect. Therefore, the explanation or description of the first aspect is also applicable to the second aspect, the third aspect, the fourth aspect, and the fifth aspect. In addition, the beneficial effects that can be achieved in the second aspect, the third aspect, the fourth aspect, and the fifth aspect can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0031] Figure 1 A schematic diagram showing an example environment in which various embodiments of the present application can be implemented;

[0032] Figure 2 A schematic diagram showing an example of synchronizing incremental operation record sets according to some embodiments of the present application;

[0033] Figure 3 A schematic diagram showing another example of synchronizing incremental operation record sets according to some embodiments of the present application;

[0034] Figure 4 A schematic flow chart of a data synchronization method according to some embodiments of the present application is shown;

[0035] Figure 5A schematic diagram showing an example of generating a hash value of an incremental operation record set according to some embodiments of the present application;

[0036] Figure 6 A block diagram of a data synchronization device according to some embodiments of the present application is shown;

[0037] Figure 7 A schematic block diagram showing an example device that can be used to implement an example implementation of the present disclosure;

[0038] Figure 8 A schematic block diagram illustrating an example device cluster that may be used to implement an example implementation of the present disclosure; and

[0039] Fig. 9 A schematic block diagram of another example device that can be used to implement an example implementation of the present disclosure is shown. . DETAILED DESCRIPTION

[0040] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0041] In the description of the embodiments of the present application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0042] As mentioned above, there are still many technical problems that need to be solved in the collaborative operation of data services between databases. In order to solve the problem of collaborative processing of data services between different databases, a traditional solution is business dual writing. When the business party performs database operations, operations are performed on multiple databases at the same time. However, the database dual writing solution brings additional costs when implementing actual business functions, such as increased response time, increased development complexity, and business impairment due to data consistency. Since the transaction management of each data type is relatively independent, the modification of the same data requires the success / failure of multiple databases at the same time. This brings inevitable difficulties to business implementation.

[0043] To avoid the above problems, another traditional solution is database synchronization. This solution is to synchronize data between different databases by subscribing to changes in the core database and updating other auxiliary databases. The database synchronization solution solves the complexity damage caused by double writing to the business. However, for specific business parties, database synchronization implementation is usually not universal, and targeted data synchronization is often performed based on specific businesses. Secondly, the solution lacks a data consistency verification method that is independent of different data types. For example, when the synchronous data receiver encounters an exception such as a rollback in the database, there is a lack of a consistency verification mechanism to promptly detect the exception and resynchronize the incremental data from the latest normal data.

[0044] In order to solve at least some of the above problems and other potential problems, in an embodiment of the present application, the synchronization manager obtains N incremental operation record sets arranged in chronological order from the incremental log of the source database, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently. Then the synchronization manager determines the target incremental operation record set from the N incremental operation record sets according to the check code of the incremental operation record set before the Nth incremental operation record set, wherein the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently. Next, the synchronization manager synchronizes the incremental operation record set after the target incremental operation record set in the N incremental operation record sets to the target database. Based on this approach, the embodiment of the present application can ensure that the previous incremental operation record set and the N incremental operation record sets are synchronized to the target database when synchronizing the Nth incremental operation record set, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies and automatically repairing data with synchronization anomalies through a consistency check method, solving abnormal problems such as rollback of the database, ensuring accurate synchronization of operations and data between databases, and improving user experience. Furthermore, the embodiment of the present application can also implement a data synchronization solution between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0045] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present application can be implemented. Figure 1 As shown, the environment 100 includes a synchronization manager 104, which is used to synchronize the incremental logs in the database 102 to the databases 106-1, 106-2, ..., and 106-M, where M is a positive integer. For ease of description, the database 102 may also be referred to as the source database 102, and the databases 106-1, 106-2, ..., and 106-M may also be referred to as the database 106 or the target database 106. Figure 1A database 102 is shown as a source database, which is only an example and not a specific limitation of the present disclosure. There may be multiple databases as source databases. Alternatively or additionally, the source database and the target database form a database system. In some embodiments, the incremental log in the target database 106 may also be synchronized to the source database 102.

[0046] In some embodiments, the data synchronization manager 104 can be deployed in a cloud management platform, or a source database, or a target database. In some embodiments, the synchronization manager 104 can be implemented in a personal computer, a server, a handheld or laptop device, a mobile device, a multiprocessor system, a consumer electronic product, a small computer, a mainframe computer, or a distributed computing environment including any one of the above systems or devices. In some embodiments, the synchronization manager 104 can include but is not limited to a personal computer, a server, a handheld or laptop device, a mobile device, a multiprocessor system, a consumer electronic product, a small computer, a mainframe computer, or a distributed computing environment including any one of the above systems or devices. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0047] When a user operates the source database 102, such as adding, deleting or modifying data in the database or one or more data tables in the database, an incremental log corresponding to the database operation will be generated and stored in the source database. For example, the incremental log records which operations are performed on the database, the operation execution time, etc. In addition, the incremental log can also record which tables in the database are operated. In order to ensure that these database operations performed in the source database 102 are also executed in the target database 106, the synchronization manager 104 needs to obtain the incremental log in the source database 102 to generate an incremental operation record set, and then send the incremental operation record in the incremental operation record set to the target database 106 to perform the corresponding incremental operation. In one example, the synchronization manager 104 requests the incremental log in the corresponding time period from the source database 102 at regular time intervals or at a predetermined time. In another example, the source database 102 can actively transmit the incremental log in a certain period or a certain number of times to the synchronization manager. The above examples are only used to describe the present disclosure, and are not specifically limited to the present disclosure. Those skilled in the art can obtain the incremental log from the source database in any suitable manner.

[0048] After acquiring the incremental log, the synchronization manager 104 will generate an incremental operation record set and then store it in a local memory. In one example, the synchronization manager 104 can store all incremental operation record sets formed by the incremental log from the source database. In another example, the synchronization manager 104 has limited storage space, and its storage space can only store a predetermined number of incremental operation record sets. At this time, only a predetermined number of incremental operation record sets within the sliding time / space window can be stored. For example, when the number of incremental operation record sets stored in the synchronization manager does not reach the predetermined number, the newly received incremental operation record set can be stored; if the number of incremental operation record sets stored reaches the predetermined number, when a new incremental operation record set is received, the longest saved incremental operation record set needs to be deleted to store the newly received incremental operation record set. Alternatively or additionally, when the synchronization manager 104 acquires the incremental operation record set, it also generates a check code for the incremental operation record set, which can be used to verify the incremental operation record set.

[0049] Next, the synchronization manager 104 will also send its locally stored incremental operation record set to the target database 106 for executing the corresponding incremental operation in the target database 106 according to the incremental operation record in the incremental operation record set (it should be noted that, for the convenience of description, the target database executing the incremental operation according to the incremental operation record is also directly referred to as the target database executing the incremental operation record in the following text of this application). When the incremental operation record set to be synchronized is to be sent from the synchronization manager 104 to the target database 106, the synchronization manager 104 needs to determine that the incremental operation record set to be synchronized has been successfully synchronized to the target incremental operation record set in the target database 106. Then, the incremental operation record set after the target incremental operation record set is synchronized to the target database 106, thereby realizing the accurate synchronization of database operations or data between the source database 102 and the target database 106.

[0050] In some embodiments, the source database 102 and the target database 106 are of different types. For different types of source databases 102 and target databases 106, the following can be combined: Figure 2 In some embodiments, the source database 102 and the target database 106 are of the same type. For the source database 102 and the target database 106 of the same type, the following can be combined: Figure 3 Description to synchronize incremental operation record sets between databases of the same type.

[0051] In some embodiments, the synchronization manager 104 can implement database-level synchronization between the source database 102 and the target database 106. For example, all incremental logs in the source database 102 are formed into an incremental operation record set, which is then sent to the target database for execution. Alternatively or additionally, when the source database 102 and the target database 106 are of the same or different types, database-level synchronization can be performed. In some embodiments, the synchronization manager 104 can implement data table-level synchronization between the source database 102 and the target database 106. At this time, only the incremental operation records for the target data table are sent to the target database. In one example, when the incremental log is obtained from the source database 102, a filter is performed, and only the incremental operation records for the target data table in the incremental log are used to form an incremental operation record set, which is then sent to the target database for execution. In another example, all incremental logs are received from the source database 102 to form an incremental operation record set, which is then stored in the synchronization manager. When the operation records in the incremental operation record set in the synchronization manager are sent to the target database, the incremental operation records related to the target data table are filtered out and sent to the target database for execution. Alternatively or additionally, when the source database 102 and the target database 106 are of the same or different types, data table-level incremental operation record synchronization may be performed. The above examples are only used to describe the present disclosure, but are not specific limitations of the present disclosure.

[0052] In this way, when synchronizing the incremental operation record set, it is ensured that the previous incremental operation record set and the record set are synchronized to the target database, so that when synchronizing the incremental operation record set in real time, it is possible to quickly discover synchronization anomalies through the consistency verification method and automatically repair the data with synchronization anomalies, solve the abnormal problems such as rollback of the database, ensure the accurate synchronization of operations and data between databases, and improve the user experience. Furthermore, the embodiment of the present application can also realize data synchronization solutions between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0053] Combined with the above Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present application can be implemented is described below. Figure 2 An example 200 for synchronizing incremental operation record sets according to some embodiments of the present application is described. Figure 2 As shown, the synchronization manager 204 is connected to a source database 202 and two target databases 206 and 208 to synchronize the incremental operation record set obtained from the source database 202 to the target databases 206 and 208 . Figure 2 This is only an example and not a specific limitation of the present disclosure. The synchronization manager 204 can synchronize incremental operation record sets between any number of source databases and target databases.

[0054] like Figure 2 As shown, the source database 202 and the target databases 206 and 208 can be databases of different types or databases of the same type. A parser 230 is configured for the source database 202 in the synchronization manager 204. When there are multiple source databases, there can be a corresponding parser for each source database. The parser 230 can obtain incremental logs from the source database 202, and then can use the incremental logs in the source database 202 to generate an incremental operation record set in a source format, for example, directly reading the incremental operation records in the source format from the multiple incremental logs obtained as the incremental operation record set in the source format. The parser 230 can also convert the incremental operation record set in the source format into an incremental operation record set in an intermediate format. In some embodiments, the parser 230 converts the incremental operation record set in the source format into an incremental operation record set in a Json format. In some embodiments, the parser 230 converts the incremental operation record set in the source format into an incremental operation record set in any suitable format that can be used as an intermediate format. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0055] like Figure 2 As shown, the parser 230 stores the acquired incremental operation record set in the synchronization manager 204 in time sequence, forming a plurality of incremental operation record sets arranged in time sequence, such as incremental operation record set t1 210, incremental operation record set t2 212, incremental operation record set t3 214, incremental operation record set t4 216, incremental operation record set t5 218, incremental operation record set t6 220, incremental operation record set t7 222, and incremental operation record set t8 224. After receiving the new incremental operation record set and converting the format, an incremental operation record set t9 226 is generated. Additionally, after the synchronization manager 204 generates the incremental operation record set in the intermediate format using the parser 230 matching the source database 202, it also generates a check code corresponding to each incremental operation record set for verifying the incremental operation record set.

[0056] For multiple incremental operation record sets arranged in chronological order, an adapter matching each target database type is used to send the incremental operation record set to the corresponding target database for execution, so as to achieve synchronization of the incremental operation record set. Figure 2 As shown, for the target database 206 , the incremental operation records in the incremental operation record set are obtained through the adapter 232 ; for the target database 208 , the incremental operation records in the incremental operation record set are obtained through the adapter 234 .

[0057] Taking the target database 206 as an example, when preparing to send the incremental operation records in the incremental operation record set t4216 to be processed to the target database 206, the adapter 232 will obtain the comparison check code from the target database 206. The comparison check code is stored in the target database 206, which is the check code of the latest incremental operation record set that has been successfully synchronized to the target database 206. Then the synchronization manager 204 searches for the same check code as the comparison check code from the check codes of the incremental operation record sets before the operation record set t4 216, for example, traverses a group of check codes of the incremental operation record sets before the incremental operation record set t4 216, and compares the comparison check code with each check code in the group of check codes. If the check code that is the same as the comparison check code is found, the incremental operation record set after the target incremental operation record set corresponding to the check code is synchronized to the target database. In one example, if all incremental operation record sets are stored in the synchronization manager 204, the synchronization manager 204 searches for the same check code as the comparison check code from the check codes of all incremental operation record sets before the incremental operation record set t4 216. Since the check codes of all incremental operation record sets are stored in the synchronization manager 204, the check code identical to the control check code can be found. In another example, if the sequence of incremental operation record sets stored in the synchronization manager 204 is a plurality of incremental operation record sets within a sliding time window, the synchronization manager 204 searches for a check code identical to the control check code from the check code of at least one incremental operation record set located within the sliding time window before the operation record set t4 216, and if the corresponding check code is found, the incremental operation record set after the target incremental operation record set corresponding to the check code is synchronized to the target database. Additionally, if the check code identical to the control check code is not found in the check codes of the plurality of incremental operation record sets within the sliding time window, all incremental operation records in the source database are sent to the target database for execution.

[0058] The adapter 232 can convert the incremental operation records in the intermediate format incremental operation record set into the target format incremental operation records suitable for the target database 206, and then send them to the target database 206 for execution. In some embodiments, the parser can also include the function of the adapter, and the adapter can also include the function of the parser to synchronize the incremental operation record set in the target database with the source database.

[0059] Figure 2The synchronization manager 204 shown can realize the synchronization of incremental operation record sets at the database level between the source database 202 and the target database 206, and can also realize the synchronization of incremental operation record sets at the data table level. In some embodiments, the parser 230 obtains the incremental operation record set in the source format from the incremental log of the source database 202, and then converts the incremental operation record set in the source format into the incremental operation record set in the intermediate format. When synchronizing to the target database 206, the adapter 232 is configured to convert the incremental operation record set into the incremental operation record set in the target format and send it to the target database 206 for execution. In some embodiments, the synchronization manager 204 realizes the synchronization of incremental operation record sets at the data table level on the parser 230 side. For example, after obtaining the incremental log from the source database 202, the incremental operation records in the incremental log are screened using the identifier of the target data table, and only the incremental operation records corresponding to the target data table form the incremental operation record set in the source format, and then the conversion is performed to form the incremental operation record set in the intermediate format. When processed by the adapter 232, the incremental operation record set in the intermediate format is converted into the incremental operation record set in the target format and then sent to the target database for execution. In some embodiments, the synchronization manager 204 implements synchronization of incremental operation record sets at the data table level on the adapter 232 side. For example, after obtaining the incremental log, the parser 230 forms an incremental operation record set in a source format based on all incremental operation records in the incremental log, and then converts the incremental operation record set in the source format into an incremental operation record set in an intermediate format. The adapter 232 can be configured by the user to filter incremental operation records in an intermediate format for one or more target data tables from the incremental operation record set, for example, based on the identifier of the target data table. The adapter 232 only converts the filtered incremental operation records in the intermediate format into incremental operation records in the target format, and then sends them to the target database for execution. Alternatively, the adapter 232 can first convert the incremental operation record set in the intermediate format into the incremental operation record set in the target format. Then, the incremental operation records for one or more target data tables are filtered to be sent to the target database for execution. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0060] Alternatively or additionally, after the incremental operation record set or the incremental operation record corresponding to the target data table is sent to the target database, if all the incremental operation records are successfully executed in the target database, the synchronization manager will obtain a response from the target database indicating that the operation corresponding to the incremental operation record is successfully executed in the target database. Then, the synchronization manager 204 sends the verification code of the incremental operation record set to the target database for storage.

[0061] In this way, when synchronizing an incremental operation record set, it is ensured that the previous incremental operation record set and the record set are synchronized to the target database, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies through the consistency verification method and automatically repairing the data with synchronization anomalies, solving abnormal problems such as rollback of the database, ensuring the accurate synchronization of operations and data between databases, and improving user experience. Furthermore, by synchronizing incremental operation record sets between different types of databases, a data synchronization solution between multiple different types of databases can be implemented.

[0062] Combined with the above Figure 2 Describes an example of synchronizing incremental operation record sets according to some embodiments of the present application. Figure 3 Another example 300 for synchronizing incremental operation record sets according to some embodiments of the present application is described. Figure 3 As shown, the synchronization manager 304 is connected to the source database 302 and the target database 306, and is used to synchronize the incremental operation record set between the source database 302 and the target database 306. Figure 3 The source database 302 and the target database 306 are databases of the same type. Therefore, the synchronization manager 304 does not need to adjust the format of the incremental operation record set from the source database 302, but can directly store it in the local storage device of the synchronization manager. Figure 3 The synchronization manager 304 shown in the figure is connected to a source database 302 and a target database 306, which is only an example and not a specific limitation of the present disclosure. The synchronization manager 304 can be connected to multiple source databases and target databases of the same type.

[0063] like Figure 3As shown, the synchronization manager 304 stores incremental operation record set t1 308, incremental operation record set t2 310, incremental operation record set t3 312, incremental operation record set t4 314, incremental operation record set t5 316, incremental operation record set t6 318, incremental operation record set t7 320, and incremental operation record set t8 322 arranged in chronological order. Incremental operation record set t9 324 is the incremental operation record set being obtained from the source database 302. Since the source database 302 and the target database 306 are of the same type, the synchronization manager does not need to adjust the format of the incremental operation records in the incremental log obtained from the source database 302, and stores the incremental operation record set t9 324 generated according to the latest acquired incremental log in the synchronization manager 304. The synchronization manager 304 also generates a checksum for the incremental operation record set t9 324. Alternatively or additionally, the synchronization manager 304 may also set a parser for each source database, and the parser is only used to obtain the incremental operation record set from the source database 302, and then store the obtained incremental operation record set in the local memory of the synchronization manager.

[0064] The ordered incremental operation record set stored in the synchronization manager 304 can be synchronized to the target database 306. When the synchronization manager 304 synchronizes the incremental operation record set to the target database, it is necessary to first determine the target incremental operation record set that has been successfully synchronized to the target database 306 before the incremental operation record set to be synchronized. Then, the incremental operation record set after the target incremental operation record set is synchronized to the target database. The method of determining the target incremental operation record set is similar to Figure 2 The target incremental operation record set is determined in the same way as described in .

[0065] In some embodiments, the synchronization manager 304 can implement the synchronization of incremental operation records at the database level. For example, the synchronization manager 304 obtains the incremental log from the source database 302, and then stores the obtained incremental log as an incremental operation record set in the local memory of the synchronization manager 304. Then the synchronization manager 304 sends the locally stored incremental operation record set to the target database 306 for execution. In some embodiments, the synchronization manager 304 can implement the synchronization of incremental operation records at the data table level. In one example, the synchronization manager 304 can implement the synchronization of incremental operation records at the data table level when obtaining incremental operation records from the source database to synchronize only the incremental operation records for certain data tables. For example, the synchronization manager 304 obtains the incremental log from the source database, and then obtains the incremental operation record corresponding to the data table from the obtained incremental log according to the identification of the data table (i.e., the target data table) to generate the incremental operation record set. In another example, the incremental operation record synchronization at the data table level can be implemented when the incremental operation record set stored locally in the synchronization manager 304 is synchronized to the target database. For example, the synchronization manager 304 can obtain the incremental operation record for the target data table from the incremental operation record set to be synchronized, such as obtaining the incremental operation record related to the target data table from the stored incremental operation record set by identifying the target data table, and then send the obtained incremental operation record related to the target data table to the target database for execution.

[0066] Alternatively or additionally, after the incremental operation record set or the incremental operation record related to the target data table is sent to the target database, if the incremental operation corresponding to the incremental operation record is successfully executed in the target database, a response indicating that the operation corresponding to the incremental operation record is successfully executed is returned to the synchronization manager. Then, the synchronization manager 304 sends the verification code of the incremental operation record set to the target database for storage.

[0067] In this way, the embodiments of the present application can ensure that the previous incremental operation record set and the latest incremental operation record set are synchronized to the target database when synchronizing the latest incremental operation record set, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies through a consistency verification method and automatically repairing the data with synchronization anomalies, and can promptly repair or restore the incremental operation record set in the target database, solving abnormal problems such as rollback of the database, ensuring operation and data synchronization between databases, and improving user experience.

[0068] Combined with the above Figure 2 and Figure 3 Describes an example for synchronizing incremental operation record sets. Figure 4 A schematic flow chart describing a data synchronization method according to some embodiments of the present application, wherein Figure 4 The method shown can be Figure 1 The synchronization manager 104 in or any suitable computing device is executed. Figure 4 The data synchronization method shown in is applied to a database system, which includes a source database and a target database.

[0069] At block 402, N incremental operation record sets arranged in chronological order are obtained from the incremental log of the source database, where N is an integer greater than 1, and the Nth incremental operation record set is the incremental operation record set currently to be synchronized. Figure 1 The synchronization manager 104 in the source database 102 obtains the incremental operation record set from the incremental log in the source database 102 and arranges them in chronological order. As mentioned above, the synchronization manager 104 can obtain the incremental operation record set from the source database at regular intervals. Accordingly, these N incremental operation record sets can be obtained in N times. The Nth incremental operation record set is the incremental operation record set to be synchronized with the target database, and the previous incremental operation record set has previously performed synchronization operations on the target database. For example, Figure 2 The incremental operation record set to be synchronized for the target database 206 is the incremental operation record set t4 216. At this time, the N incremental operation record sets arranged in chronological order are the incremental operation record set t1 210, the incremental operation record set t2 212, the incremental operation record set t3 214, and the incremental operation record set t4 216. The N incremental operation record sets for the target database 208 are the incremental operation record set t1 210, the incremental operation record set t2 212, the incremental operation record set t3 214, the incremental operation record set t4 216, the incremental operation record set t5 218, and the incremental operation record set t6 220.

[0070] In some embodiments, the synchronization manager also generates corresponding verification codes for N incremental operation record sets. For example, the synchronization manager can generate a verification code for the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set. In one example, the synchronization manager obtains a hash value by performing a hash operation on the incremental operation record set, and the hash value is used as the verification code of the incremental operation record set. For example, by using the MD5 algorithm, the SHA-1 algorithm, etc. Figure 5 A method for obtaining a hash value of an incremental operation record set is described. In another example, the synchronization manager obtains the verification code of the incremental operation record set by performing any appropriate operation on the incremental operation record set. The above examples are only used to describe the present disclosure, but are not specific limitations of the present disclosure.

[0071] In some embodiments, the source database and the target database are of different types. In order to ensure that the incremental operation records of the incremental log from the source database can be executed in the target database, a parser is set for the source database in the synchronization manager, and the parser can match the type of the incremental log in the source database and read the incremental operation records in the source format therefrom. Then, using the parser that matches the source database, the Nth source format incremental operation record set is obtained from the incremental log of the source database. For example, the parser uses a data sequence number or a timestamp to read the incremental operation record from the incremental log of the source database to generate a source format incremental operation record set. Then the Nth source format incremental operation record set is converted into the Nth intermediate format incremental operation record set. Therefore, the N incremental operation record sets stored in the synchronization manager are N intermediate format incremental operation record sets, and are arranged in chronological order. Alternatively, or additionally, when the type of the source database and the target database is the same, the source format incremental operation record set from the source database can also be converted into an intermediate format incremental operation record set.

[0072] In some embodiments, when the source database and the target database are of the same type, the incremental operation record set in the source format from the source database does not need to be converted in format, but can be directly stored in the synchronization manager.

[0073] In some embodiments, the synchronization manager obtains N sets of incremental operation records arranged in chronological order from the incremental log of the source database for the entire source database. In some embodiments, when the synchronization manager obtains N sets of incremental operation records arranged in chronological order from the incremental log of the source database, it may only obtain incremental operation records for certain data tables. The synchronization manager may obtain N sets of incremental operation records corresponding to the target data table arranged in chronological order from the incremental log of the source database according to the identifier of the target data table to be synchronized. Alternatively or additionally, regardless of whether the source database and the target database are of the same type, only data table-level synchronization may be performed. The above examples are only used to describe the present disclosure and are not specific limitations of the present disclosure.

[0074] At block 404, based on the checksum of the incremental operation record set before the Nth incremental operation record set, a target incremental operation record set is determined, where the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently. Figure 2 As shown, when the synchronization manager 204 is synchronizing the incremental operation record set t4 216 to the target database, it needs to determine the incremental operation record set that was successfully synchronized to the target database the most recently from the incremental operation record set before the incremental operation record set t4 216. The determination process is implemented using the verification code of the incremental operation record set.

[0075] In some embodiments, when determining the target incremental operation record set, the synchronization manager 204 obtains a comparison check code from the target database, which is stored in the target database and is the check code of the latest incremental operation record set that has been successfully synchronized to the target database. Usually, after the synchronization manager 204 sends the incremental operation record in an incremental operation record set to the target database for execution, it will receive a response from the target database that the incremental operation record has been successfully executed. After receiving the response that the last incremental operation record has been successfully executed, the synchronization manager will also send the check code of the incremental operation record set to the target database. The target database stores the check code in a predetermined location for easy search, such as setting a patent table to store the check code. Since the target database may be a distributed database, it includes a master database and a slave database. Usually, when the master database successfully receives the incremental operation record set and its corresponding check code successfully sent by the synchronization manager 204, it will be stored in the master database first and then downloaded to the slave database. Since it takes time to download to the slave database, the incremental operation record set and the corresponding check code received from the synchronization manager in the master storage library may not have been sent to the slave database. If the master database fails at this time, the slave database is adjusted to the master database. Since the incremental operation record set and the corresponding check code that were recently successfully transmitted to the previous master database were not received from the slave database, the incremental operation record set and its check code that were recently successfully stored in the target database at this time are not the incremental operation record set and the corresponding check code that were recently successfully sent by the synchronization manager 204. At this time, the target database has been rolled back. In order to avoid the problem caused by the rollback of the target database, the synchronization manager searches for the check code that is the same as the control check code in the check code of the incremental operation record set before the Nth incremental operation record set. If the check code that is the same as the control check code is found, the incremental operation record set corresponding to the sample check code is determined as the target incremental operation record set. In one example, before the incremental operation record in the new incremental operation record set is to be sent to the target database, the synchronization manager 204 can send a request for obtaining the check code of the latest incremental operation record set that is successfully synchronized in the target database through the adapter 232, and then receive the control check code from the target database.

[0076] At block 406, the incremental operation record set after the target incremental operation record set among the N incremental operation record sets is synchronized to the target database. Figure 2As shown, for the target database 206, when the incremental operation record set t4 216 is to be synchronized, if the synchronization manager 204 determines that the incremental operation record set t3 214 is the target incremental operation record set, the incremental operation record set t4 216 is synchronized to the target database 206. If the synchronization manager 204 determines that the incremental operation record set t2 212 is the target incremental operation record set, the incremental operation record set t3 214 and the incremental operation record set t4 216 are synchronized to the target database 206.

[0077] In some embodiments, the synchronization manager is provided with an adapter that matches the type of the target database. When synchronizing the incremental operation record set to the target database, the incremental operation record set in the Nth intermediate format is converted into the incremental operation record set in the Nth target format using the adapter that matches the type of the target database. Then the synchronization manager sends the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format. For example, when the types of the source database and the target database are different, the incremental operation record set in the source format obtained from the source database is converted into the incremental operation record set in the intermediate format for storage by the synchronization manager. Then, when the incremental operation record set stored in the synchronization manager is sent to the target database for execution, the incremental operation record set in the intermediate format is converted into the incremental operation record set in the target format that can be used for the target database. Alternatively or additionally, when the formats of the source database and the target database are the same, the incremental operation record set in the source format obtained from the source database can also be converted into the incremental operation record set in the intermediate format. Then, when sending to the target database, the incremental operation record set in the intermediate format is converted into an incremental operation record set in the target format corresponding to the target database.

[0078] In some embodiments, when the source database and the target database are of the same type, the incremental operation record set obtained from the source database may not be converted in format, but directly stored in the synchronization manager. In this case, when synchronizing the incremental synchronization operation set to the target database, the Nth incremental operation record set is directly sent to the target database to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

[0079] In some embodiments, the incremental operation records in the incremental operation record set stored in the synchronization manager can be synchronized to the target database. In some embodiments, when the incremental operation record set is synchronized to the target database, synchronization at the data table level can be performed. For example, the synchronization manager screens out the incremental operation records in the intermediate format related to the target data table from the incremental operation record set in the intermediate format of the Nth intermediate format according to the identifier of the target data table. The synchronization manager then converts the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table. The synchronization manager sends the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation records in the target format related to the target data table.

[0080] In some embodiments, when synchronizing the incremental operation record set to the target database, the synchronization manager filters out the incremental operation record related to the target data table from the Nth incremental operation record set according to the identifier of the target data table. The synchronization manager then sends the incremental operation record related to the target data table to the target database so that the target database performs the incremental operation corresponding to the incremental operation record related to the target data table. For example, when the target database is of the same type as the source database, since the incremental operation record set from the source database is directly stored in the synchronization manager, when it is sent to the target database, it is directly filtered according to the target data table identifier.

[0081] In some embodiments, after determining that the Nth incremental operation record set has been successfully synchronized to the target database, a verification code of the Nth incremental operation record set is sent to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0082] In this way, the embodiment of the present application can ensure that the previous incremental operation record set and the N record sets are synchronized to the target database when synchronizing the Nth incremental operation record set, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies and automatically repairing the data with synchronization anomalies through the consistency verification method, and can promptly repair or restore the incremental operation record set of the target database, solving abnormal problems such as rollback of the database, ensuring the operation and data synchronization between databases, and improving the user experience. Furthermore, the embodiment of the present application can also implement a data synchronization solution between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0083] Combined with the above Figure 4 A schematic flow chart of a data synchronization method according to an embodiment of the present application is described. Figure 5An example 500 of generating a hash value of an incremental operation record set according to some embodiments of the present application is described. Figure 5 As shown, when the synchronization manager obtains the incremental operation record set from the source database, it calculates the hash value of the current incremental operation record set by calculating the hash fields of the current incremental operation record set and the most recent incremental operation record set. Figure 5 As shown, at time 502 before t1, there is a corresponding incremental operation record T 512 and a hash value h 522 corresponding to the incremental operation record set T 512. Then, at time t1 504, the synchronization manager receives the incremental operation record set T1 514, and uses the formula H1=hash(hash(T1)+h) shown in box 524 to calculate the hash value H1 corresponding to the incremental operation record set T1 514. In the calculation process, the synchronization manager first calculates the intermediate hash value of the incremental operation record set T1 514, and then performs a hash operation on the intermediate hash value and the hash value of the previous incremental operation record set to obtain the hash value for the incremental operation record set T1 514. Then, at time t2 506, the incremental operation record set T2 516 is obtained, and the hash value of T2 516 is calculated using the formula H2=hash(hash(T2)+h1) in box 526; at time t3 508, the incremental operation record set T3 518 is obtained, and the hash value of the incremental operation record set T3 518 is calculated using the formula H3=hash(hash(T3)+h2) in box 528; at time t4 510, the incremental operation record set T4 520 is obtained, and the hash value of the incremental operation record set T4 520 is calculated using the formula H4=hash(hash(T4)+h2) in box 530.

[0084] When synchronization is started, the first synchronized data needs to be additionally compatible, because the data before the incremental operation record set does not have a hash value, for example, a hash operation is performed on the initial incremental operation record set to obtain a hash value for the initial incremental operation record set. In this way, the embodiment of the present application can quickly and accurately calculate a suitable hash value.

[0085] Figure 6 The data synchronization device 600 according to an embodiment of the present application is further shown. The data synchronization device 600 is applied to a database system, and the database system includes a source database and a target database. The data synchronization device 600 may include multiple modules for performing the following steps: Figure 4The corresponding steps in the method 400 discussed in . The data synchronization device 600 includes: an incremental operation record set acquisition unit 602, configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently; a target incremental operation record set determination unit 604, configured to determine the target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set, the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently; and an incremental operation record set synchronization unit 606, configured to synchronize the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

[0086] In some embodiments, the data synchronization device 600 also includes: a verification code generation unit, configured to generate a verification code for the Nth incremental operation record set; and a verification code sending unit, configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0087] In some embodiments, the target incremental operation record set determination unit 604 includes: a comparison verification code acquisition unit, configured to obtain a comparison verification code from the target database, the comparison verification code is stored in the target database, and is the verification code of the latest incremental operation record set that has been successfully synchronized to the target database; and a verification code search unit, configured to determine the target incremental operation record set by searching for a verification code that is the same as the comparison verification code in the verification code of the incremental operation record set before the Nth incremental operation record set, and the verification code of the target incremental operation record set is the same as the comparison verification code.

[0088] In some embodiments, the source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in intermediate formats, and the incremental operation record set acquisition unit 606 includes: a parser-based incremental operation record set acquisition unit, configured to use a parser that matches the type of the source database to acquire the Nth incremental operation record set in the source format from the incremental log of the source database; and a first incremental operation record set conversion unit, configured to convert the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format.

[0089] In some embodiments, the incremental operation record set synchronization unit 606 includes: a second incremental operation record set conversion unit, configured to convert the incremental operation record set in the Nth intermediate format into the incremental operation record set in the Nth target format using an adapter that matches the type of the target database; and an incremental operation record set sending unit, configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

[0090] In some embodiments, the incremental operation record set synchronization unit 606 includes: a first incremental operation record screening unit, configured to screen out incremental operation records in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table; and a first incremental operation record sending unit, configured to send the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation records in the target format related to the target data table.

[0091] In some embodiments, the incremental operation record set acquisition unit 602 includes: a data table-based incremental operation record set acquisition unit configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

[0092] In some embodiments, the incremental operation record set synchronization unit 606 includes: a second incremental operation record screening unit, configured to screen out incremental operation records related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and a second incremental operation record sending unit, configured to send the incremental operation records related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record related to the target data table.

[0093] In some embodiments, the incremental operation record set synchronization unit 606 includes: a target database synchronization unit configured to send the Nth incremental operation record set to the target database, so as to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

[0094] In some embodiments, the verification code generation unit includes: a verification code generation unit for an incremental operation record set, configured to generate a verification code for an Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

[0095] As an example of a software functional unit, the unit included in the data synchronization device may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, the data synchronization device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, a region may include multiple AZs.

[0096] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0097] As an example of a hardware functional unit, the data synchronization device and / or the synchronization manager may include at least one computing device, such as a server, etc. Alternatively, each unit of the data synchronization device and / or the synchronization manager may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0098] The multiple computing devices included in the data synchronization device and / or synchronization manager can be distributed in the same region or in different regions. The multiple computing devices included in the data synchronization device and / or synchronization manager can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data synchronization device and / or synchronization manager can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0099] The embodiment of the present disclosure further provides a computing device 700 . Figure 7 A schematic block diagram of an example device that can be used to implement an example implementation of the present disclosure is shown. Figure 7 As shown, computing device 700 includes: bus 702, processor 704, memory 706 and communication interface 708. Processor 704, memory 706 and communication interface 708 communicate through bus 702. Computing device 700 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 700. Bus 702 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7Only one line is used to represent it, but it does not mean that there is only one bus or one type of bus. Bus 704 may include a path for transmitting information between various components of computing device 700 (e.g., memory 706, processor 704, communication interface 708). Processor 704 may include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). Memory 706 may include volatile memory, such as random access memory (RAM). Processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid state drive (SSD). The memory 706 stores executable program codes, and the processor 704 executes the executable program codes to respectively implement the methods of the incremental operation record set acquisition unit 602, the target incremental operation record set determination unit 604, and the incremental operation record set synchronization unit 606 of the aforementioned embodiments of the present disclosure. That is, the memory 706 may store instructions for the methods and functions in any of the above embodiments. The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0100] The embodiment of the present disclosure further provides a computing device cluster 800 . Figure 8A schematic block diagram of an example device cluster that can be used to implement an exemplary implementation of the present disclosure is shown. According to an embodiment of the present disclosure, a synchronization manager is deployed on the computing device cluster 800, and the computing device cluster 800 includes at least one computing device 700. In some embodiments, the computing device cluster 800 may include a computing device 700 to implement all or part of the synchronization manager of an embodiment of the present disclosure. In some embodiments, the computing device cluster 800 may include a plurality of computing devices 700, and the plurality of computing devices may implement all or part of the synchronization manager of an embodiment of the present disclosure synchronously or asynchronously, serially or in parallel, independently or in a distributed manner. The computing device 700 in the computing device cluster 800 may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone. Figure 8 As shown, the computing device cluster 800 includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster 800 may store the same instructions for executing the method and function in any of the above embodiments. In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster 800 may also store partial instructions for executing the method and function of the embodiment of the present disclosure in any of the above embodiments. In other words, the combination of one or more computing devices 700 can jointly execute the instructions for the method and function of the embodiment of the present disclosure. It should be noted that the memory 706 in different computing devices 700 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the apparatus 600. That is, the instructions stored in the memory 706 in different computing devices 700 can implement the functions of one or more units or sub-units of the incremental operation record set acquisition unit 602, the target incremental operation record set determination unit 604, and the incremental operation record set synchronization unit 606.

[0101] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig. 9 1 is a schematic block diagram of another example device that can be used to implement the exemplary implementation of the present disclosure, wherein a possible implementation 900 is shown. Fig. 9 As shown, two computing devices 700A and 700B are connected via a network 910 . Fig. 9 The connection between the computing device clusters shown may be considered to be part of the functions to be performed by the computing device 700B, considering that the method of the present application needs to store a large amount of user data and perform intensive real-time or near-real-time calculations. It should be understood that Fig. 9 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices 700.

[0102] The embodiments of the present disclosure also provide a computer program product including instructions, which may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the method and function of any of the above embodiments.

[0103] The embodiments of the present disclosure also provide a computer-readable storage medium, which may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc. The computer-readable storage medium includes instructions that instruct the computing device to execute the method and function of any of the above embodiments.

[0104] In general, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be performed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented as, by way of non-limiting example, hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof.

[0105] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer executable instructions, such as instructions included in a program module, which are executed in a device on a real or virtual processor of the target to perform the process / method as described above with reference to the accompanying drawings. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of program modules can be combined or divided between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0106] The computer program code for realizing the method of the present disclosure can be written in one or more programming languages. These computer program codes can be provided to the processor of general-purpose computer, special-purpose computer or other programmable data processing device, so that the program code, when being executed by computer or other programmable data processing device, causes the function / operation specified in flow chart and / or block diagram to be implemented. The program code can be executed completely on computer, partly on computer, as independent software package, partly on computer and partly on remote computer or completely on remote computer or server.

[0107] In the context of the present disclosure, computer program codes or related data may be carried by any appropriate carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, and the like. Examples of signals may include electrical, optical, radio, acoustic or other forms of propagation signals, such as carrier waves, infrared signals, and the like.

[0108] A computer-readable medium may be any tangible medium containing or storing a program for or related to an instruction execution system, apparatus or device, or a data storage device such as a data center containing one or more available media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination thereof. More detailed examples of computer-readable storage media include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0109] In addition, although the operation of the method of the present disclosure is described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flow chart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present disclosure can be embodied in one device. Conversely, the features and functions of a device described above can be further divided into being embodied by multiple devices.

[0110] The various implementations of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those of ordinary skill in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the various embodiments of the present invention. Without departing from the scope and spirit of the various implementations described, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used in this article is intended to explain the principles of each implementation, practical application or improvement of technology in the market, or to enable other ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data synchronization method, characterized in that: The method is applied to a database system, the database system includes a source database and a target database, and the method includes: Obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized; Determine a target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set, the target incremental operation record set being the incremental operation record set that was successfully synchronized to the target database the most recently; and The incremental operation record set after the target incremental operation record set among the N incremental operation record sets is synchronized to the target database.

2. The method according to claim 1, characterized in that The method further comprises: Generating a check code for the Nth incremental operation record set; and After determining that the Nth incremental operation record set has been successfully synchronized to the target database, a verification code of the Nth incremental operation record set is sent to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

3. The method according to claim 1 or 2, characterized in that: The determining the target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set includes: Acquire a comparison check code from the target database, the comparison check code being stored in the target database and being the check code of the latest incremental operation record set that has been successfully synchronized to the target database; and The target incremental operation record set is determined by searching for a check code identical to the reference check code in the check code of the incremental operation record set before the Nth incremental operation record set, and the check code of the target incremental operation record set is identical to the reference check code.

4. The method according to any one of claims 1 to 3, characterized in that The source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in an intermediate format, and obtaining the N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: Using a parser that matches the type of the source database, obtaining an Nth source format incremental operation record set from the incremental log of the source database; and The incremental operation record set in the Nth source format is converted into an incremental operation record set in the Nth intermediate format.

5. The method according to claim 4, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: Using an adapter that matches the type of the target database, converting the incremental operation record set in the Nth intermediate format into an incremental operation record set in the Nth target format; and The incremental operation record set in the Nth target format is sent to the target database, so that the target database executes the incremental operation corresponding to the incremental operation record set in the Nth target format.

6. The method according to claim 4, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: According to the identifier of the target data table, filtering out the incremental operation records in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format; converting the incremental operation record in the intermediate format associated with the target data table into an incremental operation record in the target format associated with the target data table; and The incremental operation record in the target format related to the target data table is sent to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record in the target format related to the target data table.

7. The method according to any one of claims 1 to 3, characterized in that The step of obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: According to the identifier of the target data table, N incremental operation record sets arranged in chronological order are obtained from the incremental log of the source database.

8. The method according to any one of claims 1 to 3, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: According to the identifier of the target data table, filtering the incremental operation records related to the target data table from the Nth incremental operation record set; and The incremental operation record associated with the target data table is sent to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record associated with the target data table.

9. The method according to claim 1, characterized in that: The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: The Nth incremental operation record set is sent to the target database, so that the target database executes the incremental operation corresponding to the Nth incremental operation record set.

10. The method according to claim 2, characterized in that The generating of the verification code of the Nth incremental operation record set includes: Based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set, a verification code of the Nth incremental operation record set is generated.

11. A data synchronization device, characterized in that: The device is applied to a database system, the database system includes a source database and a target database, and the data synchronization device includes: An incremental operation record set acquisition unit is configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently; a target incremental operation record set determining unit, configured to determine a target incremental operation record set based on a check code of an incremental operation record set before the Nth incremental operation record set, the target incremental operation record set being the incremental operation record set that was most recently successfully synchronized to the target database; and The incremental operation record set synchronization unit is configured to synchronize the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

12. The data synchronization device according to claim 11, characterized in that: The data synchronization device also includes: a verification code generating unit, configured to generate a verification code for the Nth incremental operation record set; and The verification code sending unit is configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

13. The data synchronization device according to claim 11 or 12, characterized in that: The target incremental operation record set determination unit includes: a comparison check code acquisition unit, configured to acquire a comparison check code from the target database, wherein the comparison check code is stored in the target database and is a check code of the latest incremental operation record set that has been successfully synchronized to the target database; and The check code search unit is configured to determine the target incremental operation record set by searching for a check code identical to the reference check code in the check code of the incremental operation record set before the Nth incremental operation record set, wherein the check code of the target incremental operation record set is identical to the reference check code.

14. The data synchronization device according to any one of claims 11 to 13, characterized in that: The source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in an intermediate format, and the incremental operation record set acquisition unit includes: a parser-based incremental operation record set acquisition unit, configured to acquire an Nth incremental operation record set in a source format from an incremental log of the source database using a parser matching the type of the source database; and The first incremental operation record set conversion unit is configured to convert the incremental operation record set in the Nth source format into an incremental operation record set in the Nth intermediate format.

15. The data synchronization device according to claim 14, characterized in that: The incremental operation record set synchronization unit includes: a second incremental operation record set conversion unit configured to convert the incremental operation record set in the Nth intermediate format into an incremental operation record set in the Nth target format by using an adapter matching the type of the target database; and The incremental operation record set sending unit is configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

16. The data synchronization device according to claim 14, characterized in that: The incremental operation record set synchronization unit includes: A first incremental operation record screening unit is configured to screen the incremental operation record in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation record in the intermediate format related to the target data table into an incremental operation record in a target format related to the target data table; and The first incremental operation record sending unit is configured to send the incremental operation record in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record in the target format related to the target data table.

17. The data synchronization device according to any one of claims 11 to 13, characterized in that: The incremental operation record set acquisition unit includes: The data table-based incremental operation record set acquisition unit is configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

18. The data synchronization device according to any one of claims 11 to 13, characterized in that: The incremental operation record set synchronization unit includes: a second incremental operation record screening unit configured to screen the incremental operation records related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and The second incremental operation record sending unit is configured to send the incremental operation record related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record related to the target data table.

19. The data synchronization device according to claim 11, characterized in that: The incremental operation record set synchronization unit includes: The target database synchronization unit is configured to send the Nth incremental operation record set to the target database, so as to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

20. The data synchronization device according to claim 12, characterized in that: The verification code generating unit comprises: The verification code generating unit of the incremental operation record set is configured to generate the verification code of the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images