Data synchronization method, apparatus, device and storage medium

Abstract

The present application relates to the technical field of data storage, and provides a data synchronization method, an apparatus, a device and a storage medium. The method comprises: acquiring from an increment log of a source database N increment operation record sets that are arranged in a chronological order, N being an integer greater than 1, and the N-th increment operation record set being an increment operation record set to be synchronized currently. The method further comprises: on the basis of a check code of an increment operation record set previous to the N-th increment operation record set, determining a target increment operation record set, the target increment operation record set being the latest increment operation record set which is successfully synchronized to a target database. The method also comprises: synchronizing to the target database an increment operation record set subsequent to the target increment operation record set among the N increment operation record sets. When synchronizing increment operation record sets, the embodiments of the present application can quickly discover synchronization anomalies and automatically repair same, so as to realize accurate synchronization among databases, thus improving user experience.

Description

Data synchronization method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese invention patent application with application number 202310698905.X, entitled “A data synchronization method, device, equipment and storage medium” and filing date June 13, 2023, and incorporates the entirety of that application into this document by reference. Technical Field

[0003] The embodiments of the present application mainly relate to the field of data storage. More specifically, the embodiments of the present application relate to a data synchronization method, apparatus, device and storage medium. Background Art

[0004] With the rapid development of various applications, users often use databases to store data for easier management and use. Databases are used to electronically store data collections. They can contain any type of data, including text, numbers, images, videos, and documents. As user demands increase, database functionality and performance are rapidly improving. Improved database functionality and performance are crucial for organizations and users who use databases, as appropriate functionality can significantly enhance users' data processing capabilities and save time and resources.

[0005] With the development of database technology, complex business systems often use multiple storage systems. For example, one type of database can be used as a transactional database to carry out business operations, while the other type of database can be used as a full-text search and analytical database. However, when it comes to collaborative data operations between databases, there are still many technical challenges that need to be solved. For example, how to improve data synchronization performance between databases is a key technical issue.

[0006] Summary of the Invention

[0007] An embodiment of the present application provides a data synchronization solution.

[0008] According to a first aspect of the present application, a data synchronization method is provided. The method is applied to a database system, which includes a source database and a target database. The method includes: obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized; determining a target incremental operation record set based on the checksum of the incremental operation record set before the Nth incremental operation record set, where the target incremental operation record set is the incremental operation record set that was most recently successfully synchronized to the target database; and synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

[0009] In this way, when synchronizing the Nth incremental operation record set, it is ensured that the previous incremental operation record set and the N record sets are synchronized to the target database, so that when synchronizing the incremental operation record set in real time, it is possible to quickly discover synchronization anomalies through the consistency verification method and automatically repair the data with synchronization anomalies, solving abnormal problems such as rollback in the database, ensuring the accurate synchronization of operations and data between databases, and improving user experience.

[0010] In some embodiments, the method further includes: generating a checksum for the Nth incremental operation record set; and after determining that the Nth incremental operation record set has been successfully synchronized to the target database, sending the checksum for the Nth incremental operation record set to the target database, where the checksum for the Nth incremental operation record set is stored in the target database. In this manner, the checksum can be generated and stored in the target database, thereby enabling consistency verification to quickly detect synchronization anomalies and automatically repair the anomaly data.

[0011] In some embodiments, based on the checksum of the incremental operation record set before the Nth incremental operation record set, determining the target incremental operation record set includes: obtaining a reference checksum from the target database, the reference checksum stored in the target database, being the checksum of the latest incremental operation record set that has been successfully synchronized to the target database; and determining the target incremental operation record set by searching for a checksum identical to the reference checksum among the checksums of the incremental operation record set before the Nth incremental operation record set, the checksum of the target incremental operation record set being identical to the reference checksum. In this way, using the checksum, the incremental operation record set that needs to be synchronized can be quickly and accurately determined, resolving synchronization anomalies and automatically repairing data with synchronization anomalies.

[0012] In some embodiments, the source database and the target database are of different types, and the N incremental operation record sets are N incremental operation record sets in an intermediate format. Acquiring the N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: utilizing a parser that matches the type of the source database to acquire the Nth incremental operation record set in the source format from the incremental log of the source database; and converting the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format. In this manner, synchronization of incremental operation records between multiple different types of databases can be achieved.

[0013] In some embodiments, synchronizing an incremental operation record set subsequent to a target incremental operation record set among N incremental operation record sets to a target database includes: using an adapter that matches the type of the target database, converting the incremental operation record set in the Nth intermediate format into an incremental operation record set in the Nth target format; and sending the incremental operation record set in the Nth target format to the target database, so that the target database executes the incremental operation corresponding to the incremental operation record set in the Nth target format. In this manner, synchronization of incremental operation records between multiple different types of databases can be achieved.

[0014] In some embodiments, synchronizing an incremental operation record set following a target incremental operation record set from N incremental operation record sets to a target database includes: filtering out incremental operation records in an intermediate format related to the target data table from the Nth incremental operation record set in an intermediate format based on an identifier of the target data table; converting the incremental operation records in the intermediate format related to the target data table into incremental operation records in a target format related to the target data table; and sending the incremental operation records in the target format related to the target data table to the target database, so that the target database executes the incremental operation corresponding to the incremental operation record in the target format related to the target data table. In this way, synchronization of incremental operation records between data tables in multiple different types of databases can be achieved.

[0015] In some embodiments, obtaining N sets of incremental operation records arranged in chronological order from the incremental log of the source database includes: obtaining N sets of incremental operation records arranged in chronological order from the incremental log of the source database based on an identifier of the target data table. In this manner, synchronization of incremental operation records between data tables in databases of the same type can be achieved.

[0016] In some embodiments, synchronizing an incremental operation record set subsequent to a target incremental operation record set among N incremental operation record sets to a target database includes: filtering incremental operation records related to the target data table from the Nth incremental operation record set based on an identifier of the target data table; and sending the incremental operation records related to the target data table to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record related to the target data table. In this manner, synchronization of incremental operation records between data tables in databases of the same type can be achieved.

[0017] In some embodiments, synchronizing an incremental operation record set subsequent to a target incremental operation record set among N incremental operation record sets to a target database includes sending the Nth incremental operation record set to the target database, causing the target database to perform the incremental operation corresponding to the Nth incremental operation record set. In this manner, the Nth incremental operation record set can be accurately synchronized between databases.

[0018] In some embodiments, generating a verification code for the Nth incremental operation record set includes generating a verification code for the Nth incremental operation record set based on a verification code for the N-1th incremental operation record set and the Nth incremental operation record set. In this manner, verification information for the incremental operation record set can be quickly and accurately obtained for use in verifying the incremental operation record set.

[0019] According to a second aspect of the present application, a data synchronization device is provided. The device is applied to a database system, the database system including a source database and a target database, and the data synchronization device is deployed on a cloud management platform, or the source database, or the target database, and the device includes: an incremental operation record set acquisition unit, configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized; a target incremental operation record set determination unit, configured to determine a target incremental operation record set based on a checksum of an incremental operation record set before the Nth incremental operation record set, where the target incremental operation record set is the incremental operation record set that was most recently successfully synchronized to the target database; and an incremental operation record set synchronization unit, configured to synchronize the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

[0020] In some embodiments, the data synchronization device also includes: a verification code generation unit, configured to generate a verification code for the Nth incremental operation record set; and a verification code sending unit, configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0021] In some embodiments, the target incremental operation record set determination unit includes: a control verification code acquisition unit, configured to obtain a control verification code from the target database, the control verification code is stored in the target database, and is the verification code of the latest incremental operation record set that has been successfully synchronized to the target database; and a verification code search unit, configured to determine the target incremental operation record set by searching for a verification code that is the same as the control verification code in the verification code of the incremental operation record set before the Nth incremental operation record set, and the verification code of the target incremental operation record set is the same as the control verification code.

[0022] In some embodiments, the source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in an intermediate format, and the incremental operation record set acquisition unit includes: a parser-based incremental operation record set acquisition unit, configured to use a parser that matches the type of the source database to obtain the Nth incremental operation record set in the source format from the incremental log of the source database; and a first incremental operation record set conversion unit, configured to convert the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format.

[0023] In some embodiments, the incremental operation record set synchronization unit includes: a second incremental operation record set conversion unit, configured to convert the incremental operation record set in the Nth intermediate format into the incremental operation record set in the Nth target format using an adapter that matches the type of the target database; and an incremental operation record set sending unit, configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

[0024] In some embodiments, the incremental operation record set synchronization unit includes: a first incremental operation record screening unit, configured to screen out incremental operation records in the intermediate format related to the target data table from the Nth intermediate format incremental operation record set according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table; and a first incremental operation record sending unit, configured to send the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform incremental operations corresponding to the incremental operation records in the target format related to the target data table.

[0025] In some embodiments, the incremental operation record set acquisition unit includes: a data table-based incremental operation record set acquisition unit configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

[0026] In some embodiments, the incremental operation record set synchronization unit includes: a second incremental operation record screening unit, configured to screen out incremental operation records related to the target data table from the Nth incremental operation record set based on the identifier of the target data table; and a second incremental operation record sending unit, configured to send the incremental operation records related to the target data table to the target database, so as to enable the target database to perform incremental operations corresponding to the incremental operation records related to the target data table.

[0027] In some embodiments, the incremental operation record set synchronization unit includes: a target database synchronization unit configured to send the Nth incremental operation record set to the target database, so that the target database performs the incremental operation corresponding to the Nth incremental operation record set.

[0028] In some embodiments, the verification code generation unit includes: a verification code generation unit for an incremental operation record set, configured to generate a verification code for the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

[0029] According to the third aspect of the present application, a computing device cluster is also provided, comprising at least one computing device, each computing device comprising a processor and a memory, the processor of the at least one computing device being used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect of the present application.

[0030] According to a fourth aspect of the present application, a computer program product comprising instructions is further provided, wherein when the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect of the present application.

[0031] According to the fifth aspect of the present application, a computer-readable storage medium is further provided, characterized in that it includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect of the present application.

[0032] It is understandable that the apparatus of the second aspect, the computing device cluster of the third aspect, the computer program product of the fourth aspect, or the computer-readable storage medium of the fifth aspect provided above is used to execute the method provided in the first aspect. Therefore, the explanation or description of the first aspect also applies to the second, third, fourth, and fifth aspects. In addition, the beneficial effects that can be achieved in the second, third, fourth, and fifth aspects can refer to the beneficial effects in the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0034] FIG1 is a schematic diagram illustrating an example environment in which various embodiments of the present application can be implemented;

[0035] FIG2 shows a schematic diagram of an example for synchronizing incremental operation record sets according to some embodiments of the present application;

[0036] FIG3 shows a schematic diagram of another example for synchronizing incremental operation record sets according to some embodiments of the present application;

[0037] FIG4 shows a schematic flow chart of a data synchronization method according to some embodiments of the present application;

[0038] FIG5 is a schematic diagram showing an example of generating a hash value of an incremental operation record set according to some embodiments of the present application;

[0039] FIG6 shows a block diagram of a data synchronization device according to some embodiments of the present application;

[0040] FIG7 shows a schematic block diagram of an example device that can be used to implement an example implementation of the present disclosure;

[0041] FIG8 shows a schematic block diagram of an example device cluster that may be used to implement example implementations of the present disclosure; and

[0042] FIG9 shows a schematic block diagram of another example device that can be used to implement the example implementations of the present disclosure. DETAILED DESCRIPTION

[0043] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0044] In the description of the embodiments of this application, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. can refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0045] As mentioned previously, collaborative data operations between databases still present many technical challenges that need to be addressed. To enable collaborative data operations across different databases, a traditional solution is dual write operations. When executing database operations, businesses simultaneously perform operations on multiple databases. However, this dual write approach introduces additional costs when implementing actual business functions, such as increased response time, increased development complexity, and business losses due to data consistency. Because transactions for each data type are managed relatively independently, modifications to the same data require ensuring the success / failure of modifications across multiple databases. This inevitably creates difficulties in business implementation.

[0046] To avoid the aforementioned problems, another traditional solution is database synchronization. This solution synchronizes data between different databases by subscribing to changes in the core database and updating other auxiliary databases. Database synchronization solves the complexity introduced by dual writes. However, for specific business entities, database synchronization implementations are generally not universal, often requiring targeted data synchronization based on specific business needs. Furthermore, this solution lacks data consistency verification methods that are independent of different data types. For example, if an exception such as a rollback occurs on the synchronized data recipient, there is no consistency verification mechanism to promptly detect the anomaly and resynchronize the incremental data starting from the latest normal data.

[0047] In order to solve at least some of the above problems and other potential problems, in an embodiment of the present application, the synchronization manager obtains N incremental operation record sets arranged in chronological order from the incremental log of the source database, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized. Then, the synchronization manager determines the target incremental operation record set from the N incremental operation record sets based on the check code of the incremental operation record set before the Nth incremental operation record set, wherein the target incremental operation record set is the incremental operation record set that was successfully synchronized to the target database the most recently. Next, the synchronization manager synchronizes the incremental operation record set after the target incremental operation record set in the N incremental operation record sets to the target database. Based on this approach, the embodiment of the present application can ensure that the previous incremental operation record set and the N incremental operation record sets are synchronized to the target database when synchronizing the Nth incremental operation record set, so that when synchronizing the incremental operation record sets in real time, it is capable of quickly discovering synchronization anomalies and automatically repairing data with synchronization anomalies through a consistency verification method, solving abnormal problems such as rollbacks that occur in the database, ensuring accurate synchronization of operations and data between databases, and improving user experience. Furthermore, the embodiment of the present application can also implement a data synchronization solution between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0048] FIG1 shows a schematic diagram of an example environment 100 in which multiple embodiments of the present application can be implemented. As shown in FIG1 , the environment 100 includes a synchronization manager 104, which is used to synchronize the incremental logs in the database 102 to the databases 106-1, 106-2, ..., and 106-M, where M is a positive integer. For ease of description, the database 102 may also be referred to as the source database 102, and the databases 106-1, 106-2, ..., and 106-M may also be referred to as the database 106 or the target database 106. FIG1 shows a database 102 as a source database, which is merely an example and not a specific limitation of the present disclosure. There may be multiple databases serving as source databases. Alternatively or additionally, the source database and the target database form a database system. In some embodiments, the incremental logs in the target database 106 may also be synchronized to the source database 102.

[0049] In some embodiments, the data synchronization manager 104 can be deployed in a cloud management platform, a source database, or a target database. In some embodiments, the synchronization manager 104 can be implemented in a personal computer, a server, a handheld or laptop device, a mobile device, a multi-processor system, a consumer electronic product, a minicomputer, a mainframe computer, or a distributed computing environment including any of the above systems or devices. In some embodiments, the synchronization manager 104 can include, but is not limited to, a personal computer, a server, a handheld or laptop device, a mobile device, a multi-processor system, a consumer electronic product, a minicomputer, a mainframe computer, or a distributed computing environment including any of the above systems or devices. The above examples are merely illustrative of the present disclosure and are not intended to limit the present disclosure.

[0050] When a user performs an operation on source database 102, such as adding, deleting, or modifying data or one or more tables within the database, an incremental log corresponding to the database operation is generated and stored in the source database. For example, the incremental log records the database operations performed and the time the operations were performed. Additionally, the incremental log may also record which tables within the database were operated on. To ensure that database operations performed in source database 102 are also executed in target database 106, synchronization manager 104 needs to obtain the incremental logs from source database 102 to generate an incremental operation record set. The incremental operation records in the incremental operation record set are then sent to target database 106 to perform the corresponding incremental operations. In one example, synchronization manager 104 requests incremental logs for a corresponding time period from source database 102 at regular intervals or at predetermined times. In another example, source database 102 may proactively transmit incremental logs for a certain time period or a certain number of times to the synchronization manager. The above examples are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Those skilled in the art may employ any suitable method to obtain incremental logs from the source database.

[0051] After obtaining the incremental log, the synchronization manager 104 generates an incremental operation record set and then stores it in a local memory. In one example, the synchronization manager 104 can store all incremental operation record sets formed by the incremental log from the source database. In another example, the synchronization manager 104 has limited storage space, and its storage space can only store a predetermined number of incremental operation record sets. In this case, only a predetermined number of incremental operation record sets within the sliding time / space window can be stored. For example, when the number of incremental operation record sets stored in the synchronization manager does not reach the predetermined number, the newly received incremental operation record set can be stored; if the number of incremental operation record sets stored reaches the predetermined number, when a new incremental operation record set is received, the oldest saved incremental operation record set needs to be deleted to store the newly received incremental operation record set. Alternatively or additionally, when the synchronization manager 104 obtains the incremental operation record set, it also generates a check code for the incremental operation record set, which can be used to verify the incremental operation record set.

[0052] Next, the synchronization manager 104 will also send its locally stored incremental operation record set to the target database 106 for executing corresponding incremental operations in the target database 106 according to the incremental operation records in the incremental operation record set (it should be noted that, for the convenience of description, the target database executing incremental operations according to the incremental operation records will be directly referred to as the target database executing incremental operation records in the following text of this application). When the incremental operation record set to be synchronized is to be sent from the synchronization manager 104 to the target database 106, the synchronization manager 104 needs to determine whether the incremental operation record set to be synchronized has been successfully synchronized to the target incremental operation record set in the target database 106. The incremental operation record set after the target incremental operation record set is then synchronized to the target database 106, thereby achieving accurate synchronization of database operations or data between the source database 102 and the target database 106.

[0053] In some embodiments, the source database 102 and the target database 106 are of different types. For different types of source databases 102 and target databases 106, the following description of FIG. 2 can be used to synchronize incremental operation record sets between different databases. In some embodiments, the source database 102 and the target database 106 are of the same type. For the same type of source databases 102 and target databases 106, the following description of FIG. 3 can be used to synchronize incremental operation record sets between databases of the same type.

[0054] In some embodiments, the synchronization manager 104 can implement database-level synchronization between the source database 102 and the target database 106. For example, all incremental logs in the source database 102 are converted into an incremental operation record set, which is then sent to the target database for execution. Alternatively or additionally, database-level synchronization can be performed whether the source database 102 and the target database 106 are of the same or different types. In some embodiments, the synchronization manager 104 can implement table-level synchronization between the source database 102 and the target database 106. In this case, only incremental operation records for the target data table are sent to the target database. In one example, when the incremental logs are retrieved from the source database 102, they are filtered, and only the incremental operation records for the target data table in the incremental logs are used to form an incremental operation record set, which is then sent to the target database for execution. In another example, all incremental logs are received from the source database 102 to form an incremental operation record set, which is then stored in the synchronization manager. When the operation records in the incremental operation record set in the synchronization manager are sent to the target database, the incremental operation records related to the target data table are filtered and sent to the target database for execution. Alternatively or additionally, when the source database 102 and the target database 106 are of the same or different types, data table-level incremental operation record synchronization can be performed. The above examples are only used to describe the present disclosure, but are not intended to limit the present disclosure.

[0055] In this way, when synchronizing an incremental operation record set, both the previous incremental operation record set and the current record set are synchronized to the target database. This allows for the ability to quickly detect synchronization anomalies and automatically repair data with synchronization anomalies through consistency verification methods when synchronizing incremental operation record sets in real time. This resolves issues such as database rollbacks, ensures accurate synchronization of operations and data between databases, and improves the user experience. Furthermore, embodiments of the present application can also implement data synchronization solutions between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0056] The above description, in conjunction with FIG1 , shows a schematic diagram of an example environment 100 in which embodiments of the present application can be implemented. The following description, in conjunction with FIG2 , shows an example 200 for synchronizing incremental operation record sets according to some embodiments of the present application. As shown in FIG2 , a synchronization manager 204 is connected to a source database 202 and two target databases 206 and 208 to synchronize incremental operation record sets obtained from the source database 202 to the target databases 206 and 208. FIG2 is merely an example and not a specific limitation of the present disclosure. The synchronization manager 204 can synchronize incremental operation record sets between any number of source databases and target databases.

[0057] As shown in Figure 2, the source database 202 and the target databases 206 and 208 can be databases of different types or databases of the same type. A parser 230 is configured for the source database 202 in the synchronization manager 204. When there are multiple source databases, there can be a corresponding parser for each source database. The parser 230 can obtain incremental logs from the source database 202, and then use the incremental logs in the source database 202 to generate an incremental operation record set in a source format, for example, directly reading the incremental operation records in the source format from the multiple incremental logs obtained as the incremental operation record set in the source format. The parser 230 can also convert the incremental operation record set in the source format into an incremental operation record set in an intermediate format. In some embodiments, the parser 230 converts the incremental operation record set in the source format into an incremental operation record set in a Json format. In some embodiments, the parser 230 converts the incremental operation record set in the source format into an incremental operation record set in any suitable format that can be used as an intermediate format. The above examples are only used to describe the present disclosure and are not specific limitations of the present disclosure.

[0058] As shown in FIG2 , the parser 230 stores the acquired incremental operation record sets in the synchronization manager 204 in chronological order, forming a plurality of incremental operation record sets arranged in chronological order, such as incremental operation record set t1 210, incremental operation record set t2 212, incremental operation record set t3 214, incremental operation record set t4 216, incremental operation record set t5 218, incremental operation record set t6 220, incremental operation record set t7 222, and incremental operation record set t8 224. After receiving the new incremental operation record set and converting its format, the synchronization manager 204 generates incremental operation record set t9 226. Additionally, after generating the intermediate format incremental operation record set using the parser 230 that matches the source database 202, the synchronization manager 204 also generates a checksum corresponding to each incremental operation record set for use in verifying the incremental operation record set.

[0059] For multiple incremental operation record sets arranged in chronological order, adapters matching the target database types are used to send the incremental operation record sets to the corresponding target databases for execution, thereby achieving synchronization of the incremental operation record sets. As shown in Figure 2, for target database 206, the incremental operation records in the incremental operation record set are obtained through adapter 232; for target database 208, the incremental operation records in the incremental operation record set are obtained through adapter 234.

[0060] Taking the target database 206 as an example, when preparing to send the incremental operation records in the incremental operation record set t4 216 to be processed to the target database 206, the adapter 232 obtains a comparison check code from the target database 206. This comparison check code is stored in the target database 206 and is the check code of the most recent incremental operation record set that has been successfully synchronized to the target database 206. The synchronization manager 204 then searches for a check code that is identical to the comparison check code from the check codes of the incremental operation record sets prior to operation record set t4 216. For example, the synchronization manager 204 traverses a set of check codes of the incremental operation record sets prior to incremental operation record set t4 216 and compares the comparison check code with each check code in the set. If a check code that is identical to the comparison check code is found, the incremental operation record set subsequent to the target incremental operation record set corresponding to the check code is synchronized to the target database. In one example, if all incremental operation record sets are stored in the synchronization manager 204, the synchronization manager 204 searches for a check code that is identical to the comparison check code from the check codes of all incremental operation record sets prior to incremental operation record set t4 216. Since the synchronization manager 204 stores the check codes of all incremental operation record sets, a check code identical to the control check code can be found. In another example, if the sequence of incremental operation record sets stored in the synchronization manager 204 is a plurality of incremental operation record sets within a sliding time window, the synchronization manager 204 searches for a check code identical to the control check code from the check codes of at least one incremental operation record set located within the sliding time window before the operation record set t4 216. If a corresponding check code is found, the incremental operation record set following the target incremental operation record set corresponding to the check code is synchronized to the target database. Additionally, if a check code identical to the control check code is not found among the check codes of the plurality of incremental operation record sets within the sliding time window, all incremental operation records in the source database are sent to the target database for execution.

[0061] The adapter 232 can convert the incremental operation records in the intermediate format incremental operation record set into the target format of the target database 206, and then send the incremental operation records to the target database 206 for execution. In some embodiments, the parser can also include the functionality of the adapter, and the adapter can also include the functionality of the parser to synchronize the incremental operation record set in the target database with the source database.

[0062] The synchronization manager 204 shown in FIG2 can synchronize incremental operation records at the database level between the source database 202 and the target database 206, or at the data table level. In some embodiments, the parser 230 obtains incremental operation records in a source format from the incremental log of the source database 202 and then converts the source format incremental operation records into an intermediate format incremental operation records. When synchronizing to the target database 206, the adapter 232 is configured to convert the incremental operation records into a target format incremental operation records and send it to the target database 206 for execution. In some embodiments, the synchronization manager 204 synchronizes incremental operation records at the data table level on the parser 230 side. For example, after obtaining the incremental log from the source database 202, the incremental operation records in the incremental log are filtered using the identifier of the target data table, forming only the incremental operation records corresponding to the target data table into the source format incremental operation records. The incremental operation records are then converted to form an intermediate format incremental operation records. When processed by the adapter 232, the intermediate format incremental operation records are converted into the target format incremental operation records and then sent to the target database for execution. In some embodiments, the synchronization manager 204 implements synchronization of incremental operation record sets at the data table level on the adapter 232 side. For example, after obtaining the incremental log, the parser 230 forms an incremental operation record set in the source format based on all the incremental operation records in the incremental log, and then converts the incremental operation record set in the source format into an incremental operation record set in the intermediate format. The adapter 232 can be configured by the user to filter incremental operation records in the intermediate format for one or more target data tables from the incremental operation record set, for example, based on the identifier of the target data table. The adapter 232 only converts the filtered incremental operation records in the intermediate format into incremental operation records in the target format, and then sends them to the target database for execution. Alternatively, the adapter 232 can first convert the incremental operation record set in the intermediate format into the incremental operation record set in the target format. Then, the incremental operation records for one or more target data tables are filtered to be sent to the target database for execution. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0063] Alternatively or additionally, after sending the incremental operation record set or the incremental operation record corresponding to the target data table to the target database, if all incremental operation records are successfully executed in the target database, the synchronization manager will obtain a response from the target database indicating that the operation corresponding to the incremental operation record is successfully executed in the target database. The synchronization manager 204 then sends a verification code of the incremental operation record set to the target database for storage.

[0064] This approach ensures that both the previous incremental operation record set and the new record set are synchronized to the target database when synchronizing an incremental operation record set. This allows for real-time synchronization of incremental operation record sets, enabling the rapid detection of synchronization anomalies through consistency verification and automatic repair of these anomalies. This resolves issues like database rollbacks, ensures accurate synchronization of operations and data between databases, and improves the user experience. Furthermore, by synchronizing incremental operation record sets between different types of databases, a data synchronization solution between multiple database types can be implemented.

[0065] An example for synchronizing incremental operation record sets according to some embodiments of the present application is described above in conjunction with Figure 2. Another example 300 for synchronizing incremental operation record sets according to some embodiments of the present application is described below in conjunction with Figure 3. As shown in Figure 3, a synchronization manager 304 is connected to a source database 302 and a target database 306 to synchronize incremental operation record sets between the source database 302 and the target database 306. In Figure 3, the source database 302 and the target database 306 are databases of the same type. Therefore, the synchronization manager 304 does not need to adjust the format of the incremental operation record sets from the source database 302, but can directly store them in the local storage device of the synchronization manager. The synchronization manager 304 shown in Figure 3 is connected to one source database 302 and one target database 306, which is only an example and not a specific limitation of the present disclosure. The synchronization manager 304 can be connected to multiple source databases and target databases of the same type.

[0066] As shown in Figure 3, the synchronization manager 304 stores incremental operation record sets t1 308, t2 310, t3 312, t4 314, t5 316, t6 318, t7 320, and t8 322, arranged in chronological order. Incremental operation record set t9 324 is the incremental operation record set currently being retrieved from source database 302. Because source database 302 and target database 306 are of the same type, the synchronization manager does not need to format the incremental operation records in the incremental log retrieved from source database 302. Instead, the synchronization manager stores incremental operation record set t9 324, generated based on the most recently retrieved incremental log, in the synchronization manager 304. The synchronization manager 304 also generates a checksum for incremental operation record set t9 324. Alternatively or additionally, the synchronization manager 304 may also set a parser for each source database, and the parser is only used to obtain the incremental operation record set from the source database 302, and then store the obtained incremental operation record set in the local memory of the synchronization manager.

[0067] The ordered incremental operation record sets stored in the synchronization manager 304 can be synchronized to the target database 306. When synchronizing the incremental operation record sets to the target database, the synchronization manager 304 must first determine a target incremental operation record set that has been successfully synchronized to the target database 306 and precedes the incremental operation record set to be synchronized. The incremental operation record set following the target incremental operation record set is then synchronized to the target database. The method for determining the target incremental operation record set is the same as that described in FIG. 2 .

[0068] In some embodiments, the synchronization manager 304 can implement database-level incremental operation record synchronization. For example, the synchronization manager 304 obtains incremental logs from the source database 302 and then stores the obtained incremental logs as an incremental operation record set in the local memory of the synchronization manager 304. The synchronization manager 304 then sends the locally stored incremental operation record set to the target database 306 for execution. In some embodiments, the synchronization manager 304 can implement table-level incremental operation record synchronization. In one example, the synchronization manager 304 can implement table-level incremental operation record synchronization when obtaining incremental operation records from the source database, so as to synchronize only the incremental operation records for certain tables. For example, the synchronization manager 304 obtains incremental logs from the source database and then, based on the identifier of the data table to be synchronized (i.e., the target data table), obtains the incremental operation records corresponding to the data table from the obtained incremental logs to generate the incremental operation record set. In another example, table-level incremental operation record synchronization can be implemented when synchronizing the locally stored incremental operation record set in the synchronization manager 304 to the target database. For example, the synchronization manager 304 can obtain the incremental operation records for the target data table from the incremental operation record set to be synchronized, such as by obtaining the incremental operation records related to the target data table from the stored incremental operation record set based on the identifier of the target data table. The obtained incremental operation records related to the target data table are then sent to the target database for execution.

[0069] Alternatively or additionally, after the incremental operation record set or the incremental operation record associated with the target data table is sent to the target database, if the incremental operation corresponding to the incremental operation record is successfully executed in the target database, a response indicating that the operation corresponding to the incremental operation record is successfully executed is returned to the synchronization manager. The synchronization manager 304 then sends a verification code of the incremental operation record set to the target database for storage.

[0070] In this way, the embodiments of the present application can ensure that the previous incremental operation record set and the latest incremental operation record set are synchronized to the target database when synchronizing the latest incremental operation record set, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies through the consistency verification method and automatically repairing the data with synchronization anomalies, and can promptly repair or restore the incremental operation record set in the target database, solving abnormal problems such as rollback of the database, ensuring operation and data synchronization between databases, and improving user experience.

[0071] An example for synchronizing incremental operation record sets has been described above in conjunction with Figures 2 and 3 . A schematic flowchart of a data synchronization method according to some embodiments of the present application is described below in conjunction with Figure 4 . The method illustrated in Figure 4 can be executed by synchronization manager 104 in Figure 1 or any suitable computing device. The data synchronization method illustrated in Figure 4 is applied to a database system comprising a source database and a target database.

[0072] At box 402, N incremental operation record sets arranged in chronological order are obtained from the incremental log of the source database, where N is an integer greater than 1, and the Nth incremental operation record set is the incremental operation record set currently to be synchronized. For example, the synchronization manager 104 in Figure 1 obtains the incremental operation record set from the incremental log in the source database 102 and arranges it in chronological order. As mentioned above, the synchronization manager 104 can obtain the incremental operation record set from the source database at regular time intervals. Accordingly, these N incremental operation record sets can be obtained N times. The Nth incremental operation record set is the incremental operation record set to be synchronized with the target database, and the previous incremental operation record set has previously performed synchronization operations on the target database. For example, in FIG2 , the incremental operation record set currently to be synchronized for target database 206 is incremental operation record set t4 216. At this time, the N incremental operation record sets arranged in chronological order are incremental operation record set t1 210, incremental operation record set t2 212, incremental operation record set t3 214, and incremental operation record set t4 216. The N incremental operation record sets for target database 208 are incremental operation record set t1 210, incremental operation record set t2 212, incremental operation record set t3 214, incremental operation record set t4 216, incremental operation record set t5 218, and incremental operation record set t6 220.

[0073] In some embodiments, the synchronization manager will also generate corresponding check codes for N incremental operation record sets. For example, the synchronization manager can generate a check code for the Nth incremental operation record set based on the check code of the N-1th incremental operation record set and the Nth incremental operation record set. In one example, the synchronization manager obtains a hash value by performing a hash operation on the incremental operation record set, and the hash value is used as the check code for the incremental operation record set. For example, by using the MD5 algorithm, the SHA-1 algorithm, etc. The method for obtaining the hash value of the incremental operation record set will be described below in conjunction with Figure 5. In another example, the synchronization manager obtains its check code by performing any suitable operation on the incremental operation record set. The above examples are only used to describe the present disclosure and are not specific limitations of the present disclosure.

[0074] In some embodiments, the source database and the target database are of different types. To ensure that the incremental operation records from the incremental log of the source database can be executed in the target database, a parser is set for the source database in the synchronization manager. The parser can match the type of the incremental log in the source database and read the incremental operation records in the source format from it. Then, using the parser that matches the source database, the Nth incremental operation record set in the source format is obtained from the incremental log of the source database. For example, the parser uses a data sequence number or timestamp to read the incremental operation records from the incremental log of the source database to generate an incremental operation record set in the source format. The Nth incremental operation record set in the source format is then converted into the Nth incremental operation record set in the intermediate format. Therefore, the N incremental operation record sets stored in the synchronization manager are N incremental operation record sets in the intermediate format and are arranged in chronological order. Alternatively, or additionally, when the source database and the target database are of the same type, the incremental operation record set in the source format from the source database can also be converted into an incremental operation record set in the intermediate format.

[0075] In some embodiments, when the source database and the target database are of the same type, the incremental operation record set in the source format from the source database does not need to be format converted. Instead, the incremental operation record set in the source format can be directly stored in the synchronization manager.

[0076] In some embodiments, the synchronization manager obtains N sets of incremental operation records arranged in chronological order from the incremental log of the source database for the entire source database. In some embodiments, when the synchronization manager obtains N sets of incremental operation records arranged in chronological order from the incremental log of the source database, it may only obtain incremental operation records for certain data tables. The synchronization manager may obtain N sets of incremental operation records corresponding to the target data table to be synchronized from the incremental log of the source database according to the identifier of the target data table to be synchronized. Alternatively or additionally, regardless of whether the source database and the target database are of the same type, only data table-level synchronization may be performed. The above examples are only used to describe the present disclosure and are not specific limitations of the present disclosure.

[0077] At block 404, a target incremental operation record set is determined based on the checksum of the incremental operation record set preceding the Nth incremental operation record set. The target incremental operation record set is the incremental operation record set that was most recently successfully synchronized to the target database. As shown in FIG2 , when synchronization manager 204 synchronizes incremental operation record set t4 216 to the target database, it needs to determine the incremental operation record set that was most recently successfully synchronized to the target database from among the incremental operation record sets preceding incremental operation record set t4 216. This determination is accomplished using the checksum of the incremental operation record set.

[0078] In some embodiments, when determining the target incremental operation record set, the synchronization manager 204 obtains a comparison verification code from the target database. This comparison verification code is stored in the target database and represents the verification code of the most recent incremental operation record set successfully synchronized to the target database. Typically, after the synchronization manager 204 sends the incremental operation records in an incremental operation record set to the target database for execution, it receives a response from the target database confirming the successful execution of the incremental operation record. After receiving the response confirming the successful execution of the last incremental operation record, the synchronization manager also sends the verification code for that incremental operation record set to the target database. The target database stores this verification code in a predetermined location for easy access, such as by setting up a proprietary table to store the verification code. Since the target database may be a distributed database, it may include a master database and slave databases. Typically, when the master database successfully receives the incremental operation record set and its corresponding verification code sent by the synchronization manager 204, they are first stored in the master database and then downloaded to the slave database. Because downloading to the slave database takes time, the incremental operation record set and its corresponding verification code received by the synchronization manager from the master database may not have been sent to the slave database yet. If the master database fails at this time, the slave database is adjusted to become the master database. Since the slave database has not received the incremental operation record set and the corresponding check code that were recently successfully transmitted to the previous master database, the incremental operation record set and its check code that are now successfully stored in the target database are not the incremental operation record set and the corresponding check code that were recently successfully sent by the synchronization manager 204. At this point, the target database rolls back. In order to avoid problems caused by the rollback of the target database, the synchronization manager searches for a check code that is identical to the control check code in the check codes of the incremental operation record set before the Nth incremental operation record set. If a check code that is identical to the control check code is found, the incremental operation record set corresponding to the check code is determined as the target incremental operation record set. In one example, before the synchronization manager 204 sends the incremental operation records in the new incremental operation record set to the target database, it can send a request to the target database through the adapter 232 to obtain the check code of the latest incremental operation record set that has been successfully synchronized in the target database, and then receive the control check code from the target database.

[0079] At block 406, the incremental operation record sets following the target incremental operation record set among the N incremental operation record sets are synchronized to the target database. As shown in FIG2 , for target database 206, when synchronizing incremental operation record set t4 216, if synchronization manager 204 determines that incremental operation record set t3 214 is the target incremental operation record set, then incremental operation record set t4 216 is synchronized to target database 206. If synchronization manager 204 determines that incremental operation record set t2 212 is the target incremental operation record set, then both incremental operation record set t3 214 and incremental operation record set t4 216 are synchronized to target database 206.

[0080] In some embodiments, the synchronization manager is provided with an adapter that matches the type of the target database. When synchronizing the incremental operation record set to the target database, the adapter that matches the type of the target database is used to convert the Nth incremental operation record set in the intermediate format into the Nth incremental operation record set in the target format. The synchronization manager then sends the Nth incremental operation record set in the target format to the target database so that the target database can execute the incremental operation corresponding to the Nth incremental operation record set in the target format. For example, when the types of the source database and the target database are different, the incremental operation record set in the source format obtained from the source database is converted into an intermediate format incremental operation record set for storage by the synchronization manager. Then, when the incremental operation record set stored in the synchronization manager is sent to the target database for execution, the intermediate format incremental operation record set is converted into an incremental operation record set in the target format that can be used for the target database. Alternatively or additionally, when the formats of the source database and the target database are the same, the source format incremental operation record set obtained from the source database can also be converted into an intermediate format incremental operation record set. Then, when sending to the target database, the incremental operation record set in the intermediate format is converted into an incremental operation record set in the target format corresponding to the target database.

[0081] In some embodiments, when the source and target databases are of the same type, the incremental operation record sets obtained from the source database may be directly stored in the synchronization manager without undergoing format conversion. In this case, when synchronizing the incremental synchronization operation sets to the target database, the Nth incremental operation record set is directly sent to the target database, causing the target database to execute the incremental operation corresponding to the Nth incremental operation record set.

[0082] In some embodiments, the incremental operation records in the incremental operation record set stored in the synchronization manager can be synchronized to the target database. In some embodiments, when synchronizing the incremental operation record set to the target database, synchronization at the data table level can be performed. For example, the synchronization manager filters out the incremental operation records in the intermediate format related to the target data table from the incremental operation record set in the intermediate format according to the identifier of the target data table. The synchronization manager then converts the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table. The synchronization manager sends the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation records in the target format related to the target data table.

[0083] In some embodiments, when synchronizing an incremental operation record set to a target database, the synchronization manager filters out the incremental operation records related to the target data table from the Nth incremental operation record set based on the identifier of the target data table. The synchronization manager then sends the incremental operation records related to the target data table to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record related to the target data table. For example, when the target database is of the same type as the source database, since the incremental operation record set from the source database is directly stored in the synchronization manager, when it is sent to the target database, it is directly filtered based on the identifier of the target data table.

[0084] In some embodiments, after determining that the Nth incremental operation record set has been successfully synchronized to the target database, the verification code of the Nth incremental operation record set is sent to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0085] In this way, the embodiment of the present application can ensure that the previous incremental operation record set and the N record sets are synchronized to the target database when synchronizing the Nth incremental operation record set, so that when synchronizing the incremental operation record set in real time, it is capable of quickly discovering synchronization anomalies and automatically repairing the data with synchronization anomalies through the consistency verification method, and can promptly repair or restore the incremental operation record set of the target database, solving abnormal problems such as rollbacks that occur in the database, ensuring operation and data synchronization between databases, and improving user experience. Furthermore, the embodiment of the present application can also implement data synchronization solutions between multiple different types of databases by synchronizing incremental operation record sets between different types of databases.

[0086] The above describes a schematic flow chart of a data synchronization method according to an embodiment of the present application in conjunction with FIG4 . Below, in conjunction with FIG5 , describes an example 500 of generating a hash value for an incremental operation record set according to some embodiments of the present application. As shown in FIG5 , when the synchronization manager obtains an incremental operation record set from the source database, it calculates the hash value of the newly added incremental operation record set by calculating the hash fields of the current incremental operation record set and the most recent incremental operation record set. As shown in FIG5 , at time 502 before t1, there is a corresponding incremental operation record T 512 and a hash value h 522 corresponding to the incremental operation record set T 512. Then, at time t1 504, the synchronization manager receives the incremental operation record set T1 514 and calculates the hash value H1 corresponding to the incremental operation record set T1 514 using the formula H1 = hash(hash(T1) + h) shown in block 524. During the calculation process, the synchronization manager first calculates an intermediate hash value for the incremental operation record set T1 514, and then performs a hash operation on the intermediate hash value and the hash value of the previous incremental operation record set to obtain a hash value for the incremental operation record set T1 514. Then, at time t2 506, the incremental operation record set T2 516 is obtained, and the hash value of T2 516 is calculated using the formula H2=hash(hash(T2)+h1) in block 526. At time t3 508, the incremental operation record set T3 518 is obtained, and the hash value of the incremental operation record set T3 518 is calculated using the formula H3=hash(hash(T3)+h2) in block 528. At time t4 510, the incremental operation record set T4 520 is obtained, and the hash value of the incremental operation record set T4 520 is calculated using the formula H4=hash(hash(T4)+h2) in block 530.

[0087] When synchronization is started, the first piece of data synchronized needs to be extra compatible because the data before the incremental operation record set does not have a hash value. For example, a hash operation is performed on the initial incremental operation record set to obtain a hash value for the initial incremental operation record set. In this way, the embodiments of the present application can quickly and accurately calculate the appropriate hash value.

[0088] FIG6 further illustrates a data synchronization apparatus 600 according to an embodiment of the present application, which is applied to a database system including a source database and a target database. The data synchronization apparatus 600 may include multiple modules for executing corresponding steps in the method 400 discussed in FIG4 . The data synchronization apparatus 600 includes: an incremental operation record set acquisition unit 602 configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized; a target incremental operation record set determination unit 604 configured to determine a target incremental operation record set based on a checksum of an incremental operation record set preceding the Nth incremental operation record set, the target incremental operation record set being the incremental operation record set most recently successfully synchronized to the target database; and an incremental operation record set synchronization unit 606 configured to synchronize the incremental operation record sets following the target incremental operation record set among the N incremental operation record sets to the target database.

[0089] In some embodiments, the data synchronization device 600 also includes: a verification code generation unit, configured to generate a verification code for the Nth incremental operation record set; and a verification code sending unit, configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

[0090] In some embodiments, the target incremental operation record set determination unit 604 includes: a control verification code acquisition unit, configured to obtain a control verification code from the target database, the control verification code is stored in the target database, and is the verification code of the latest incremental operation record set that has been successfully synchronized to the target database; and a verification code search unit, configured to determine the target incremental operation record set by searching for a verification code that is the same as the control verification code in the verification code of the incremental operation record set before the Nth incremental operation record set, and the verification code of the target incremental operation record set is the same as the control verification code.

[0091] In some embodiments, the source database and the target database are of different types, and the N incremental operation record sets are N incremental operation record sets in an intermediate format. The incremental operation record set acquisition unit 606 includes: a parser-based incremental operation record set acquisition unit, configured to use a parser that matches the type of the source database to obtain the Nth incremental operation record set in the source format from the incremental log of the source database; and a first incremental operation record set conversion unit, configured to convert the Nth incremental operation record set in the source format into the Nth incremental operation record set in the intermediate format.

[0092] In some embodiments, the incremental operation record set synchronization unit 606 includes: a second incremental operation record set conversion unit, configured to convert the incremental operation record set in the Nth intermediate format into the incremental operation record set in the Nth target format using an adapter that matches the type of the target database; and an incremental operation record set sending unit, configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

[0093] In some embodiments, the incremental operation record set synchronization unit 606 includes: a first incremental operation record screening unit, configured to screen out incremental operation records in the intermediate format related to the target data table from the Nth intermediate format incremental operation record set according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation records in the intermediate format related to the target data table into incremental operation records in the target format related to the target data table; and a first incremental operation record sending unit, configured to send the incremental operation records in the target format related to the target data table to the target database, so as to enable the target database to perform incremental operations corresponding to the incremental operation records in the target format related to the target data table.

[0094] In some embodiments, the incremental operation record set acquisition unit 602 includes: a data table-based incremental operation record set acquisition unit configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

[0095] In some embodiments, the incremental operation record set synchronization unit 606 includes: a second incremental operation record screening unit, configured to screen out incremental operation records related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and a second incremental operation record sending unit, configured to send the incremental operation records related to the target data table to the target database, so as to enable the target database to perform incremental operations corresponding to the incremental operation records related to the target data table.

[0096] In some embodiments, the incremental operation record set synchronization unit 606 includes: a target database synchronization unit configured to send the Nth incremental operation record set to the target database, so as to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

[0097] In some embodiments, the verification code generation unit includes: a verification code generation unit for an incremental operation record set, configured to generate a verification code for the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

[0098] The units included in the data synchronization device are taken as an example of a software functional unit. The data synchronization device may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, the data synchronization device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Generally, a region may include multiple AZs.

[0099] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0100] As an example of a hardware functional unit, the data synchronization device and / or synchronization manager may include at least one computing device, such as a server. Alternatively, each unit of the data synchronization device and / or synchronization manager may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0101] The multiple computing devices included in the data synchronization apparatus and / or synchronization manager can be distributed in the same region or in different regions. The multiple computing devices included in the data synchronization apparatus and / or synchronization manager can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the data synchronization apparatus and / or synchronization manager can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0102] An embodiment of the present disclosure also provides a computing device 700. FIG7 shows a schematic block diagram of an example device that can be used to implement an exemplary implementation of the present disclosure. As shown in FIG7 , the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700. The bus 702 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, FIG7 shows only one line, but this does not mean that there is only one bus or one type of bus. The bus 704 can include a path for transmitting information between the various components of the computing device 700 (e.g., the memory 706, the processor 704, and the communication interface 708). The processor 704 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). The memory 706 may include a volatile memory, such as a random access memory (RAM). The processor 704 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 706 stores executable program code, and the processor 704 executes the executable program code to respectively implement the methods of the incremental operation record set acquisition unit 602, the target incremental operation record set determination unit 604, and the incremental operation record set synchronization unit 606 of the aforementioned embodiments of the present disclosure. That is, the memory 706 may store instructions for the methods and functions in any of the above embodiments. The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0103] Embodiments of the present disclosure also provide a computing device cluster 800. FIG8 shows a schematic block diagram of an example device cluster that can be used to implement an exemplary implementation of the present disclosure. A synchronization manager according to an embodiment of the present disclosure is deployed on the computing device cluster 800, which includes at least one computing device 700. In some embodiments, the computing device cluster 800 may include one computing device 700 to implement all or part of the synchronization manager of the embodiment of the present disclosure. In some embodiments, the computing device cluster 800 may include multiple computing devices 700, which may implement all or part of the synchronization manager of the embodiment of the present disclosure synchronously or asynchronously, serially or in parallel, independently or in a distributed manner. The computing devices 700 in the computing device cluster 800 may be servers, such as central servers, edge servers, or local servers in a local data center. In some embodiments, the computing devices may also be terminal devices such as desktop computers, laptop computers, or smartphones. As shown in FIG8 , the computing device cluster 800 includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster 800 may store the same instructions for executing the methods and functions of any of the above-described embodiments. In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster 800 may also respectively store partial instructions for executing the methods and functions of the embodiments of the present disclosure in any of the above-mentioned embodiments. In other words, the combination of one or more computing devices 700 can jointly execute the instructions for the methods and functions of the embodiments of the present disclosure. It should be noted that the memories 706 in different computing devices 700 in the computing device cluster can store different instructions, which are respectively used to execute partial functions of the device 600. That is, the instructions stored in the memories 706 in different computing devices 700 can realize the functions of one or more units or sub-units of the incremental operation record set acquisition unit 602, the target incremental operation record set determination unit 604 and the incremental operation record set synchronization unit 606.

[0104] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG9 shows a schematic block diagram of another example device that can be used to implement an exemplary implementation of the present disclosure, wherein a possible implementation 900 is shown. As shown in FIG9 , two computing devices 700A and 700B are connected via a network 910. The connection method between the computing device clusters shown in FIG9 may be to consider that the method of the present application requires the storage of a large amount of user data and the performance of intensive real-time or near real-time calculations, and therefore consider handing over part of the functions to the computing device 700B for execution. It should be understood that the functions of the computing device 700A shown in FIG9 may also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be performed by multiple computing devices 700.

[0105] The embodiments of the present disclosure further provide a computer program product including instructions, which may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the methods and functions of any of the above embodiments.

[0106] Embodiments of the present disclosure also provide a computer-readable storage medium, which can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the methods and functions of any of the above embodiments.

[0107] In general, various embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other pictorial representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented as, by way of non-limiting example, hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0108] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which are executed in a device on a real or virtual processor of a target to perform the process / method as described above with reference to the accompanying drawings. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided between program modules as needed. The machine-executable instructions for the program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0109] The computer program code for implementing the disclosed method can be written in one or more programming languages. These computer program codes can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code, when executed by the computer or other programmable data processing device, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on a computer, partially on a computer, as an independent software package, partially on a computer and partially on a remote computer or entirely on a remote computer or server.

[0110] In the context of the present disclosure, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, acoustic, or other forms of propagated signals, such as carrier waves, infrared signals, and the like.

[0111] A computer-readable medium may be any tangible medium containing or storing a program for or relating to an instruction execution system, apparatus, or device, or a data storage device such as a data center containing one or more available media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of computer-readable storage media include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0112] In addition, although the operations of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps. It should also be noted that the features and functions of two or more devices according to the present disclosure can be embodied in one device. Conversely, the features and functions of a device described above can be further divided into being embodied by multiple devices.

[0113] The various implementations of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed implementations. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those of ordinary skill in the art that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not deviate from the essence of the corresponding technical solutions from the scope of protection of the technical solutions of the various embodiments of the present invention. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope and spirit of the various implementations described. The choice of terms used herein is intended to explain the principles of the various implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data synchronization method, characterized in that: The method is applied to a database system, the database system includes a source database and a target database, and the method includes: Obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set currently to be synchronized; Determine a target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set, the target incremental operation record set being the incremental operation record set that was successfully synchronized to the target database the most recently; and The incremental operation record set after the target incremental operation record set among the N incremental operation record sets is synchronized to the target database.

2. The method according to claim 1, characterized in that The method further comprises: Generating a check code for the Nth incremental operation record set; and After determining that the Nth incremental operation record set has been successfully synchronized to the target database, a verification code of the Nth incremental operation record set is sent to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

3. The method according to claim 1 or 2, characterized in that: The determining the target incremental operation record set based on the check code of the incremental operation record set before the Nth incremental operation record set includes: Acquire a comparison check code from the target database, the comparison check code is stored in the target database and is the check code of the latest incremental operation record set that has been successfully synchronized to the target database; and The target incremental operation record set is determined by searching for a check code identical to the reference check code in the check code of the incremental operation record set before the Nth incremental operation record set, and the check code of the target incremental operation record set is identical to the reference check code.

4. The method according to any one of claims 1 to 3, characterized in that The source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in an intermediate format, and obtaining the N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: Using a parser that matches the type of the source database, obtaining an Nth source format incremental operation record set from the incremental log of the source database; and The incremental operation record set in the Nth source format is converted into an incremental operation record set in the Nth intermediate format.

5. The method according to claim 4, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: Using an adapter that matches the type of the target database, converting the incremental operation record set in the Nth intermediate format into an incremental operation record set in the Nth target format; and The incremental operation record set in the Nth target format is sent to the target database, so that the target database executes the incremental operation corresponding to the incremental operation record set in the Nth target format.

6. The method according to claim 4, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: According to the identifier of the target data table, filtering out the incremental operation records in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format; converting the incremental operation record in the intermediate format associated with the target data table into an incremental operation record in the target format associated with the target data table; and The incremental operation record in the target format related to the target data table is sent to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record in the target format related to the target data table.

7. The method according to any one of claims 1 to 3, characterized in that The step of obtaining N incremental operation record sets arranged in chronological order from the incremental log of the source database includes: According to the identifier of the target data table, N incremental operation record sets arranged in chronological order are obtained from the incremental log of the source database.

8. The method according to any one of claims 1 to 3, characterized in that The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: According to the identifier of the target data table, filtering the incremental operation records related to the target data table from the Nth incremental operation record set; and The incremental operation record associated with the target data table is sent to the target database, so that the target database performs the incremental operation corresponding to the incremental operation record associated with the target data table.

9. The method according to claim 1, characterized in that: The step of synchronizing the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database includes: The Nth incremental operation record set is sent to the target database, so that the target database executes the incremental operation corresponding to the Nth incremental operation record set.

10. The method according to claim 2, characterized in that The generating of the verification code of the Nth incremental operation record set includes: Based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set, a verification code of the Nth incremental operation record set is generated.

11. A data synchronization device, characterized in that: The device is applied to a database system, the database system includes a source database and a target database, and the data synchronization device includes: An incremental operation record set acquisition unit is configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database, where N is an integer greater than 1, wherein the Nth incremental operation record set is the incremental operation record set to be synchronized currently; a target incremental operation record set determining unit, configured to determine a target incremental operation record set based on a check code of an incremental operation record set before the Nth incremental operation record set, the target incremental operation record set being the incremental operation record set that was most recently successfully synchronized to the target database; and The incremental operation record set synchronization unit is configured to synchronize the incremental operation record set after the target incremental operation record set among the N incremental operation record sets to the target database.

12. The data synchronization device according to claim 11, characterized in that: The data synchronization device also includes: a verification code generating unit, configured to generate a verification code for the Nth incremental operation record set; and The verification code sending unit is configured to send the verification code of the Nth incremental operation record set to the target database after determining that the Nth incremental operation record set has been successfully synchronized to the target database, and the verification code of the Nth incremental operation record set is used to be stored in the target database.

13. The data synchronization device according to claim 11 or 12, characterized in that: The target incremental operation record set determination unit includes: a comparison check code acquisition unit, configured to acquire a comparison check code from the target database, wherein the comparison check code is stored in the target database and is a check code of the latest incremental operation record set that has been successfully synchronized to the target database; and The check code search unit is configured to determine the target incremental operation record set by searching for a check code identical to the reference check code in the check code of the incremental operation record set before the Nth incremental operation record set, wherein the check code of the target incremental operation record set is identical to the reference check code.

14. The data synchronization device according to any one of claims 11 to 13, characterized in that: The source database and the target database are of different types, the N incremental operation record sets are N incremental operation record sets in an intermediate format, and the incremental operation record set acquisition unit includes: a parser-based incremental operation record set acquisition unit, configured to acquire an Nth incremental operation record set in a source format from an incremental log of the source database using a parser matching the type of the source database; and The first incremental operation record set conversion unit is configured to convert the incremental operation record set in the Nth source format into an incremental operation record set in the Nth intermediate format.

15. The data synchronization device according to claim 14, characterized in that: The incremental operation record set synchronization unit includes: a second incremental operation record set conversion unit configured to convert the incremental operation record set in the Nth intermediate format into an incremental operation record set in the Nth target format by using an adapter matching the type of the target database; and The incremental operation record set sending unit is configured to send the incremental operation record set in the Nth target format to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record set in the Nth target format.

16. The data synchronization device according to claim 14, characterized in that: The incremental operation record set synchronization unit includes: A first incremental operation record screening unit is configured to screen the incremental operation record in the intermediate format related to the target data table from the Nth incremental operation record set in the intermediate format according to the identifier of the target data table; a data table incremental operation record conversion unit, configured to convert the incremental operation record in the intermediate format related to the target data table into an incremental operation record in a target format related to the target data table; and The first incremental operation record sending unit is configured to send the incremental operation record in a target format related to the target data table to the target data table. The target database is sent to the target database to enable the target database to execute the incremental operation corresponding to the incremental operation record in the target format related to the target data table.

17. The data synchronization device according to any one of claims 11 to 13, characterized in that: The incremental operation record set acquisition unit includes: The data table-based incremental operation record set acquisition unit is configured to acquire N incremental operation record sets arranged in chronological order from the incremental log of the source database according to the identifier of the target data table.

18. The data synchronization device according to any one of claims 11 to 13, characterized in that: The incremental operation record set synchronization unit includes: a second incremental operation record screening unit configured to screen the incremental operation records related to the target data table from the Nth incremental operation record set according to the identifier of the target data table; and The second incremental operation record sending unit is configured to send the incremental operation record related to the target data table to the target database, so as to enable the target database to perform the incremental operation corresponding to the incremental operation record related to the target data table.

19. The data synchronization device according to claim 11, characterized in that: The incremental operation record set synchronization unit includes: The target database synchronization unit is configured to send the Nth incremental operation record set to the target database, so as to enable the target database to perform the incremental operation corresponding to the Nth incremental operation record set.

20. The data synchronization device according to claim 12, characterized in that: The verification code generating unit comprises: The verification code generating unit of the incremental operation record set is configured to generate the verification code of the Nth incremental operation record set based on the verification code of the N-1th incremental operation record set and the Nth incremental operation record set.

21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

22. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images