Data synchronization method and device, equipment and medium

Abstract

The embodiment of the invention provides a data synchronization method and device, equipment and a medium, and relates to the technical field of data processing. When a source database and a target database are in a data synchronization state, the data synchronization method provided by the embodiment of the invention comprises the following steps: firstly, determining a first dynamic data table of the source database and a second dynamic data table of the target database; and then, according to the data synchronization identifier, identifying to-be-synchronized write operation data which is not synchronized to the second dynamic data table from the first dynamic data table, and only performing data synchronization on the to-be-synchronized write operation data. Therefore, according to the data synchronization method provided by the embodiment of the invention, only the to-be-synchronized write operation data can be selectively synchronized, unnecessary data transmission is avoided, and a calculation-intensive task for performing de-duplication processing is also avoided, so that the occupation of calculation resources is reduced, the time of a data synchronization process is shortened, and the data synchronization efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, device and medium. Background Art

[0002] Data synchronization is the process of ensuring that data between different databases remains consistent, which is crucial for achieving cross-database data sharing.

[0003] In related technologies, to achieve data synchronization, the last synchronization point in time can be determined by querying the synchronization logs between the source and target databases. Then, incremental data (i.e., data that has changed since the last synchronization) is extracted from the source database based on the synchronization point in time. Finally, the incremental data is deduplicated based on all data in the source database, and the deduplicated incremental data is synchronized to the target database.

[0004] However, deduplication of incremental data is a computationally intensive task, which not only makes the data synchronization process time-consuming, but also increases the load on the database system and occupies more computing resources. Summary of the Invention

[0005] Based on the above problems, the present application provides a data synchronization method, device, equipment and medium, which can reduce the occupation of computing resources, shorten the time of the data synchronization process, and improve the efficiency of data synchronization.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, the present application discloses a data synchronization method, the method comprising:

[0008] When a source database and a target database are in a data synchronization state, determining a first dynamic data table of the source database and a second dynamic data table of the target database, wherein the first dynamic data table and the second dynamic data table include write operation data, each of the write operation data includes a data synchronization flag, and the data synchronization flag indicates whether the write operation data has been synchronized;

[0009] identifying, according to the data synchronization identifier, from the first dynamic data table the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table;

[0010] Synchronize the write operation data to be synchronized to the target database.

[0011] Optionally, identifying, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier, includes:

[0012] If the comparison mode is determined to be sampling comparison according to the comparison mode field value in the first dynamic data table, and the number of rows in the first dynamic data table and the number of rows in the second dynamic data table are the same, determining a sampling step size;

[0013] determining, from the first dynamic data table, sampled write operation data and a row number of the sampled write operation data according to the sampling step size;

[0014] Determining comparison write operation data in the second dynamic data table, wherein a row number of the comparison write operation data is the same as a row number of the sample write operation data;

[0015] If the sampled write operation data and the compared write operation data are inconsistent, the sampled write operation data is identified as write operation data to be synchronized.

[0016] Optionally, identifying, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier, includes:

[0017] If the comparison mode is determined to be a full comparison according to the comparison mode field value in the first dynamic data table, the first dynamic data table and the second dynamic data table are respectively paginated to obtain a paginated first dynamic data table and a paginated second dynamic data table;

[0018] Sorting the paginated first dynamic data table and the paginated second dynamic data table in ascending order of primary keys to obtain a sorted first dynamic data table and a sorted second dynamic data table;

[0019] If the primary key of the Mth data in the sorted first dynamic data table is consistent with the primary key of the Mth data in the sorted second dynamic data table and the other fields of the Mth data in the sorted first dynamic data table are inconsistent with the other fields of the Mth data in the sorted second dynamic data table, then the Mth data in the sorted first dynamic data table is identified as the data to be synchronized write operation.

[0020] Optionally, the method further includes:

[0021] If the primary key of the Mth data in the sorted first dynamic data table is smaller than the primary key of the Mth data in the sorted second dynamic data table and the primary key of the Mth data in the sorted first dynamic data table does not exist in the primary key set corresponding to the sorted second dynamic data table, then the Mth data in the sorted first dynamic data table is identified as the data to be synchronized write operation.

[0022] Optionally, the method further includes:

[0023] When the source database and the target database are in a data synchronization state, if a data synchronization anomaly is detected, and the cause of the anomaly is a network connection failure and / or a storage device failure, the data synchronization state is terminated.

[0024] In a second aspect, the present application discloses a data synchronization device, the device comprising: a determination module, an identification module and a synchronization module;

[0025] The determining module is configured to, when the source database and the target database are in a data synchronization state, determine a first dynamic data table of the source database and a second dynamic data table of the target database, wherein the first dynamic data table and the second dynamic data table include write operation data, each of the write operation data includes a data synchronization flag, and the data synchronization flag indicates whether the write operation data has been synchronized;

[0026] The identification module is configured to identify, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier;

[0027] The synchronization module is used to synchronize the write operation data to be synchronized to the target database.

[0028] Optionally, the identification module specifically includes: a first identification submodule, a second identification submodule, a third identification submodule and a fourth identification submodule;

[0029] The first identification submodule is configured to determine a sampling step size if the comparison mode is determined to be sampling comparison according to the comparison mode field value in the first dynamic data table, and the number of rows in the first dynamic data table and the number of rows in the second dynamic data table are the same;

[0030] The second identification submodule is configured to determine the sampled write operation data and the row number of the sampled write operation data from the first dynamic data table according to the sampling step size;

[0031] The third identification submodule is configured to determine the comparison write operation data in the second dynamic data table, wherein the row number of the comparison write operation data is the same as the row number of the sample write operation data;

[0032] The fourth identification submodule is configured to identify the sampled write operation data as write operation data to be synchronized if the sampled write operation data is inconsistent with the compared write operation data.

[0033] Optionally, the identification module specifically includes: a fifth identification submodule, a sixth identification submodule and a seventh identification submodule;

[0034] The fifth identification submodule is configured to, if the comparison mode is determined to be a full comparison according to the comparison mode field value in the first dynamic data table, perform paging processing on the first dynamic data table and the second dynamic data table respectively to obtain a paginated first dynamic data table and a paginated second dynamic data table;

[0035] The sixth identification submodule is configured to sort the paginated first dynamic data table and the paginated second dynamic data table in ascending order of primary keys to obtain a sorted first dynamic data table and a sorted second dynamic data table;

[0036] The seventh identification submodule is used to identify that the Mth data in the sorted first dynamic data table is data to be synchronized with the write operation if the primary key of the Mth data in the sorted first dynamic data table is consistent with the primary key of the Mth data in the sorted second dynamic data table and the other fields of the Mth data in the sorted first dynamic data table are inconsistent with the other fields of the Mth data in the sorted second dynamic data table.

[0037] Optionally, the apparatus further comprises: an eighth identification submodule;

[0038] The eighth identification submodule is used to identify that the Mth data in the sorted first dynamic data table is the data to be synchronized write operation if the primary key of the Mth data in the sorted first dynamic data table is smaller than the primary key of the Mth data in the sorted second dynamic data table and the primary key of the Mth data in the sorted first dynamic data table does not exist in the primary key set corresponding to the sorted second dynamic data table.

[0039] Optionally, the device further comprises: a termination module;

[0040] The termination module is used to terminate the data synchronization state when the source database and the target database are in the data synchronization state and if a data synchronization anomaly is detected, and the cause of the anomaly is a network connection failure and / or a storage device failure.

[0041] In a third aspect, the present application discloses a data synchronization device, the device comprising: a memory and a processor;

[0042] The memory is used to store programs;

[0043] The processor is used to execute the program to implement the various steps of the data synchronization method as described in the first aspect.

[0044] In a fourth aspect, the present application discloses a computer-readable medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the data synchronization method described in the first aspect are implemented.

[0045] Compared with the existing technology, this application has the following beneficial effects:

[0046] The embodiments of the present application provide a data synchronization method, apparatus, equipment and medium, and relate to the field of data processing technology. When the source database and the target database are in a data synchronization state, the data synchronization method provided by the embodiments of the present application first determines the first dynamic data table of the source database and the second dynamic data table of the target database. Subsequently, according to the data synchronization identifier, the write operation data to be synchronized that has not yet been synchronized to the second dynamic data table is identified from the first dynamic data table, and data synchronization is performed only on the write operation data to be synchronized. Therefore, the data synchronization method provided by the embodiments of the present application can not only selectively synchronize only the write operation data to be synchronized, avoiding unnecessary data transmission, but also avoids the computationally intensive task of performing deduplication processing, thereby reducing the occupation of computing resources, shortening the time of the data synchronization process, and improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of another data synchronization method provided in an embodiment of the present application;

[0050] Figure 3A A flow chart of a sampling comparison provided in an embodiment of the present application;

[0051] Figure 3B A flowchart of a full comparison provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a data synchronization device provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a computer-readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] First, let’s explain the technical terms involved in this application:

[0055] A source database is a database containing data that needs to be synchronized with another database. During data synchronization or migration, data from the source database is read and then transferred to another database (the target database) based on pre-defined rules or conditions.

[0056] A target database is a database that receives data synchronized from another database (the source database). During data synchronization or migration, the target database receives data from the source database and inserts, updates, or merges this data into its existing data structure as needed.

[0057] As described above, to synchronize data, you first query the synchronization logs between the source and target databases to determine the last synchronization point in time. Then, based on the synchronization point in time, you extract incremental data from the source database (i.e., data that has changed since the last synchronization). Finally, you deduplicate this incremental data based on all data in the source database and synchronize this deduplicated incremental data to the target database.

[0058] However, deduplication of incremental data is a computationally intensive task, which not only makes the data synchronization process time-consuming, but also increases the load on the database system and occupies more computing resources.

[0059] After research, the inventors have proposed a data synchronization method, device, equipment and medium, which relate to the field of data processing technology. When the source database and the target database are in a data synchronization state, the data synchronization method provided by the embodiment of the present application first determines the first dynamic data table of the source database and the second dynamic data table of the target database. Subsequently, according to the data synchronization identifier, the write operation data to be synchronized that has not yet been synchronized to the second dynamic data table is identified from the first dynamic data table, and data synchronization is performed only on the write operation data to be synchronized. Therefore, the data synchronization method provided by the embodiment of the present application can not only selectively synchronize only the write operation data to be synchronized, avoiding unnecessary data transmission, but also avoids the computationally intensive task of deduplication processing, thereby reducing the occupation of computing resources, shortening the time of the data synchronization process, and improving the efficiency of data synchronization.

[0060] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0061] See also Figure 1 , which is a flow chart of a data synchronization method provided by an embodiment of the present application. The method includes:

[0062] S101: When a source database and a target database are in a data synchronization state, determine a first dynamic data table of the source database and a second dynamic data table of the target database, wherein the first dynamic data table and the second dynamic data table include write operation data, and each write operation data includes a data synchronization identifier, which indicates whether the write operation data has been synchronized.

[0063] The data synchronization status indicates whether the data synchronization process between the source database and the target database is in progress or has just been completed.

[0064] The first dynamic data table is a data table in the source database, which is used to record all write operation data that need to be synchronized. The second dynamic data table is a data table in the target database, which is used to receive write operation data synchronized from the source database. Write operation data refers to the data generated by operations such as insert, update or delete that occur in the source database. Each write operation data includes a data synchronization identifier, which is a flag bit or field used to indicate whether a certain write operation data has been successfully synchronized from the source database to the target database. For example, if the value of the data synchronization identifier is 0, it means that it is not synchronized, and if the value of the data synchronization identifier is 1, it means that it is synchronized. Through the data synchronization identifier, it is possible to distinguish which write operation data has been synchronized and which write operation data still needs to be synchronized, thereby avoiding duplicate synchronization and missed synchronization.

[0065] Specifically, when the source database and the target database have established a data synchronization mechanism and the data synchronization status is normal, the first dynamic data table is first searched and determined in the source database and the second dynamic data table is searched and determined in the target database.

[0066] S102: Identify, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier.

[0067] In a specific implementation, if the data synchronization flag indicates that the write operation data is not synchronized (for example, sync_flag = 0, the sync_flag field is used to identify whether the write operation data has been synchronized to the target database. For example, sync_flag = 0 indicates not synchronized, and sync_flag = 1 indicates synchronized), the write operation data is directly identified as write operation data to be synchronized that has not yet been synchronized to the second dynamic data table.

[0068] In another specific implementation, if the data synchronization flag indicates that the write operation data has been synchronized (for example, sync_flag = 1), the write operation data may also be write operation data to be synchronized. This may be due to the following reasons: First, during the network transmission process, the write operation data may be lost or incompletely transmitted to the target database. Second, the target database may have a fault when receiving the write operation data, resulting in the write operation data not being correctly written. Therefore, for the write synchronization data whose data synchronization flag indicates that the write operation data has been synchronized, additional checks are required to confirm whether it is actually synchronized to the target database.

[0069] Specifically, the consistency of the write operation data can be ensured by comparing the corresponding records in the source and target databases; the synchronization log can also be checked to confirm whether the write operation data synchronization operation has been successfully completed. This application does not limit the specific synchronization method.

[0070] S103: Synchronize the write operation data to be synchronized to the target database.

[0071] Use an appropriate transport protocol (such as TCP / IP or HTTP) to transfer the data to be synchronized from the source database to the target database.

[0072] It should be noted that to ensure the reliability and security of data transmission, encrypted transmission methods can also be used. For example, the SSL / TLS protocol can be used to encrypt the transmitted data for synchronization write operations to prevent it from being intercepted or tampered with during transmission. Alternatively, the transmitted data for synchronization write operations can be digitally signed to ensure its integrity and source authenticity.

[0073] It should also be noted that if the data synchronization flag of the write operation data to be synchronized indicates that the write operation data is not synchronized, it is necessary to update the data synchronization flag in the source database after the data is successfully synchronized to the target database, and mark the synchronized data as synchronized (for example, change sync_flag to 1).

[0074] In summary, the embodiment of the present application provides a data synchronization method. When the source database and the target database are in a data synchronization state, the data synchronization method provided by the embodiment of the present application first determines the first dynamic data table of the source database and the second dynamic data table of the target database. Subsequently, according to the data synchronization identifier, the write operation data to be synchronized that has not yet been synchronized to the second dynamic data table is identified from the first dynamic data table, and data synchronization is performed only on the write operation data to be synchronized. Therefore, the data synchronization method provided by the embodiment of the present application can not only selectively synchronize only the write operation data to be synchronized, avoiding unnecessary data transmission, but also avoids the computationally intensive task of deduplication processing, thereby reducing the occupation of computing resources, shortening the time of the data synchronization process, and improving the efficiency of data synchronization.

[0075] See also Figure 2 , which is a flow chart of another data synchronization method provided by an embodiment of the present application. The method is applied to a data synchronization device, which is connected to a source database and a target database. The method includes:

[0076] S201: The data synchronization device records the write operation data of the source database by turning on the data change recording switch.

[0077] The data change logging switch is a control option used to enable or disable the data change logging feature. By turning on the data change logging switch and activating the data change logging feature, you can ensure that all write operations to the source database are recorded for subsequent data synchronization.

[0078] When the data change logging switch is turned on, the data synchronization device begins to monitor write operations on the source database and writes the corresponding write operation data to the data synchronization device's buffer. The write operation data includes the write operation message, the transaction type (such as insert, update, delete), and the timestamp.

[0079] It should be noted that the data synchronization device needs to set a data synchronization flag for each write operation data in the buffer. The initial value of the data synchronization flag is "unsynchronized", indicating that the write operation data has not yet been synchronized to the target database.

[0080] S202: The data synchronization device synchronizes the write operation data in the first dynamic data table of the source database to the second dynamic data table of the target database.

[0081] The write operation data is transmitted from the first animation data table of the source database to the second animation data table of the target database using an appropriate transmission protocol (such as TCP / IP, HTTP, etc.).

[0082] It should be noted that if a piece of write operation data is synchronized successfully, the data synchronization flag of the write operation data in the buffer is updated to "synchronized", for example, by changing the sync_flag from 0 (unsynchronized) to 1 (synchronized).

[0083] It should also be noted that the data synchronization device must first enable the synchronization starter and stopper before synchronizing the write operation data to the target database through the message recorder. The synchronization starter and stopper is a control option used to enable or disable the data synchronization function.

[0084] S203: The data synchronization device detects whether an abnormality occurs in the data synchronization process. If not, execute S204.

[0085] The data synchronization device's exception listener needs to continuously monitor the execution of the data synchronization process to check for any errors or exceptions during execution. The exception listener is a monitoring module responsible for continuously monitoring the execution of the data synchronization process to ensure the reliability and stability of data synchronization.

[0086] In one example, if a basic resource anomaly occurs, the data synchronization process needs to be stopped immediately. Basic resource anomalies include network connection anomalies (whether there is a disconnection or high latency), storage device anomalies (whether the disk is full or damaged, etc.), etc.

[0087] In another example, if a business exception occurs (whether there are business logic errors, data inconsistencies, etc. during data synchronization), corresponding processing operations need to be performed, such as retrying synchronization, recording logs, and alarms, and continuing the synchronization process.

[0088] If the data synchronization device does not detect any abnormality in the data synchronization process, step S204 is executed.

[0089] S204: The data synchronization device identifies, from the first dynamic data table according to the data synchronization identifier, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table.

[0090] In a specific implementation, if the data synchronization flag indicates that the write operation data is not synchronized, the write operation data is directly identified as write operation data to be synchronized that has not yet been synchronized to the second dynamic data table.

[0091] In another specific implementation, if the data synchronization flag indicates that the write operation data has been synchronized, first, based on the comparison mode field value in the first dynamic data table, it is determined whether to perform a sampling comparison or a full comparison.

[0092] See also Figure 3A, which is a flow chart of a sampling comparison provided by an embodiment of the present application. Sampling comparison is to select a portion of the write operation data from the first dynamic data table randomly or according to specific rules for comparison, rather than comparing all the write operation data. Using sampling comparison, only part of the write operation data needs to be compared, which greatly reduces the time and resource consumption of the comparison. For example, assuming that there is a dynamic data table containing 1 million records for sampling comparison, 1 record can be selected from every 1,000 records for comparison, so only 1,000 records need to be compared.

[0093] In some specific implementations, if the verification mode is "sampling comparison", the specific process is as follows:

[0094] A1: Check whether the number of rows in the first and second animation tables is the same. If so, proceed to step A2.

[0095] The row count consistency check ensures that the data volume in the first and second animation tables is the same. If the row counts are the same, proceed to step A2. If not, you may need to clean up or synchronize the data to ensure the data volume in the first and second animation tables is the same, or select another verification mode.

[0096] A2: Calculate the sampling step size.

[0097] The sampling step length N is calculated based on the predetermined sampling comparison ratio or number and the number of rows (i.e., the number of data items) in the first and second dynamic data tables. Specifically, the sampling step length N = number of data items / (number of sampling items + 1). The number of sampling items is determined based on the predetermined sampling comparison ratio or number and represents the number of samples to be drawn. The number of data items is the number of rows in the first and second dynamic data tables.

[0098] It should be noted that if the number of data items is less than or equal to the sampling number M, an error message will be directly prompted, indicating that the amount of data is too small to perform effective sampling comparison.

[0099] A3: Determine the sample write operation data of the Nth, 2Nth, ..., MNth rows from the first dynamic data table.

[0100] In the first dynamic data table, data in rows N, 2N, ..., MN are determined as sampled write operation data according to the calculated sampling step N. It should be noted that the data synchronization mark of the sampled write operation data indicates that the sampled write operation data has completed synchronization.

[0101] A4: Determine the comparison write operation data in the second dynamic data table, wherein the row number of the comparison write operation data is the same as the row number of the sampling write operation data.

[0102] In the second dynamic data table, data with the same row number as the sampled write operation data is found as comparison write operation data.

[0103] A5: If the sampled write operation data and the compared write operation data are inconsistent, the sampled write operation data is identified as the write operation data to be synchronized.

[0104] Compare the sampled write operation data with the comparison write operation data one by one. If the sampled write operation data and the comparison write operation data are inconsistent, identify the sampled write operation data as the write operation data to be synchronized. If the data with the corresponding sequence number in the source database is consistent with the sampled data, it can be considered that this part of the data has been synchronized (or does not need to be synchronized), and the current data synchronization verification ends.

[0105] It should be noted that the above steps need to be performed continuously until all data in the source and target databases (or all sample data selected according to the sampling step) are traversed. This means that for each sampling point, steps A3 to A5 need to be performed until all sampling points are checked.

[0106] See also Figure 3B , this figure is a flow chart of a full comparison provided in an embodiment of the present application. The full comparison is to compare all the write operation data in the first dynamic data table one by one to ensure that the write operation data in each row is consistent. The full comparison can ensure the consistency of all write operation data and will not miss any problems. For example, suppose there is a first dynamic data table containing 1 million records. When performing a full comparison, all 1 million records need to be compared one by one.

[0107] In some specific implementations, if the verification mode is "full comparison", the specific process is as follows:

[0108] B1: Perform paging processing on the first dynamic data table and the second dynamic data table to obtain the paginated first dynamic data table and the paginated second dynamic data table.

[0109] Because the first dynamic data table may contain a large amount of write operation data, directly performing a full comparison may cause memory overflow. Therefore, it is necessary to page the first and second dynamic data tables, loading only a portion of the write operation data for comparison at a time. Specifically, the first and second dynamic data tables can be divided into multiple smaller pages (or batches), each containing a certain number of data records.

[0110] B2: Sort the paginated first dynamic data table and the paginated second dynamic data table in ascending order of the primary key to obtain a sorted first dynamic data table and a sorted second dynamic data table.

[0111] In order to simplify the comparison process and improve the comparison efficiency, it is necessary to sort the first dynamic data table and the second dynamic data table after paging in ascending order of the primary key. Specifically, a sorting algorithm (such as quick sort, merge sort, etc.) can be used to sort the first dynamic data table and the second dynamic data table after paging to obtain the sorted first dynamic data table and the sorted second dynamic data table.

[0112] B3: Determine whether the primary keys of the Mth data in the sorted first dynamic data table and the Mth data in the sorted second dynamic data table are consistent, where M is a positive integer. If yes, proceed to step B4; if no, proceed to step B5.

[0113] B4: Determine whether the other fields of the Mth data in the sorted first dynamic data table are consistent with the other fields of the Mth data in the sorted second dynamic data table. If not, execute step B6.

[0114] If the primary keys are consistent, compare the non-primary key field values ​​(i.e., other fields) of the Mth data in the sorted first dynamic data table and the sorted second dynamic data table one by one. If all fields are consistent, the data is considered synchronized; if any field is inconsistent, execute step B6.

[0115] B5: Determine whether the primary key of the Mth data item in the sorted first dynamic data table is smaller than the primary key of the Mth data item in the sorted second dynamic data table, and whether the primary key of the Mth data item in the sorted first dynamic data table does not exist in the primary key set corresponding to the sorted second dynamic data table. If so, proceed to step B6.

[0116] If the primary key of the Mth data in the sorted first dynamic data table is smaller than the primary key of the Mth data in the sorted second dynamic data table, and the primary key does not exist in the sorted second dynamic data table, it is considered that there may be a problem with the data insertion order or data loss, and step B6 is executed.

[0117] B6: Identify the Mth data in the sorted first dynamic data table as the data to be synchronized with the write operation.

[0118] When data inconsistency or data loss is found, the Mth data in the sorted first dynamic data table is identified as data to be synchronized with the write operation.

[0119] S205: The data synchronization device synchronizes the write operation data to be synchronized in the first dynamic data table of the source database to the second dynamic data table of the target database.

[0120] It should be noted that the data synchronization device must first enable the synchronization compensator before synchronizing the pending write operation data in the first dynamic data table of the source database to the second dynamic data table of the target database. The synchronization compensator is primarily used to perform data compensation synchronization on single or batch data after data synchronization to prevent data inconsistencies between the two databases due to unsuccessful synchronization of data between the source database and the target database.

[0121] In summary, the embodiment of the present application provides a data synchronization method. When the source database and the target database are in a data synchronization state, the data synchronization method provided by the embodiment of the present application first determines the first dynamic data table of the source database and the second dynamic data table of the target database. Subsequently, according to the data synchronization identifier, the write operation data to be synchronized that has not yet been synchronized to the second dynamic data table is identified from the first dynamic data table, and data synchronization is performed only on the write operation data to be synchronized. Therefore, the data synchronization method provided by the embodiment of the present application can not only selectively synchronize only the write operation data to be synchronized, avoiding unnecessary data transmission, but also avoids the computationally intensive task of deduplication processing, thereby reducing the occupation of computing resources, shortening the time of the data synchronization process, and improving the efficiency of data synchronization.

[0122] See also Figure 4 , which is a schematic diagram of a data synchronization device provided by an embodiment of the present application. The data synchronization device 400 includes: a determination module 401, an identification module 402 and a synchronization module 403.

[0123] Determining module 401, configured to, when the source database and the target database are in a data synchronization state, determine a first dynamic data table of the source database and a second dynamic data table of the target database, wherein the first dynamic data table and the second dynamic data table include write operation data, each write operation data includes a data synchronization flag, and the data synchronization flag indicates whether the write operation data has been synchronized;

[0124] An identification module 402 is configured to identify, from the first dynamic data table, data to be synchronized that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier;

[0125] The synchronization module 403 is used to synchronize the write operation data to be synchronized to the target database.

[0126] In a specific implementation, the identification module 402 specifically includes: a first identification submodule, a second identification submodule, a third identification submodule, and a fourth identification submodule;

[0127] A first identification submodule is configured to determine a sampling step size if the comparison mode is confirmed to be a sampling comparison according to the comparison mode field value in the first dynamic data table and the number of rows in the first dynamic data table and the number of rows in the second dynamic data table are the same;

[0128] A second identification submodule is configured to determine the sampled write operation data and the row number of the sampled write operation data from the first dynamic data table according to the sampling step size;

[0129] a third identification submodule, configured to determine the comparison write operation data in the second dynamic data table, wherein the row number of the comparison write operation data is the same as the row number of the sampling write operation data;

[0130] The fourth identification submodule is configured to identify the sampled write operation data as write operation data to be synchronized if the sampled write operation data and the compared write operation data are inconsistent.

[0131] In a specific implementation, the identification module 402 specifically includes: a fifth identification submodule, a sixth identification submodule, and a seventh identification submodule;

[0132] a fifth identification submodule, configured to, if the comparison mode is determined to be a full comparison according to the comparison mode field value in the first dynamic data table, perform paging processing on the first dynamic data table and the second dynamic data table respectively, to obtain a paginated first dynamic data table and a paginated second dynamic data table;

[0133] a sixth identification submodule, configured to sort the paginated first dynamic data table and the paginated second dynamic data table in ascending order of the primary key to obtain a sorted first dynamic data table and a sorted second dynamic data table;

[0134] The seventh identification submodule is used to identify that the Mth data in the sorted first dynamic data table is the data to be synchronized write operation if the primary key of the Mth data in the sorted first dynamic data table is consistent with the primary key of the Mth data in the sorted second dynamic data table and the other fields of the Mth data in the sorted first dynamic data table are inconsistent with the other fields of the Mth data in the sorted second dynamic data table.

[0135] In a specific implementation, the data synchronization device 400 further includes: an eighth identification submodule;

[0136] The eighth identification submodule is used to identify that the Mth data in the sorted first dynamic data table is the data to be synchronized write operation if the primary key of the Mth data in the sorted first dynamic data table is smaller than the primary key of the Mth data in the sorted second dynamic data table and the primary key of the Mth data in the sorted first dynamic data table does not exist in the primary key set corresponding to the sorted second dynamic data table.

[0137] In a specific implementation, the data synchronization device 400 further includes: a termination module;

[0138] The termination module is used to terminate the data synchronization state when the source database and the target database are in the data synchronization state and a data synchronization anomaly is detected, and the anomaly is caused by a network connection failure and / or a storage device failure.

[0139] In summary, an embodiment of the present application provides a data synchronization device. When a source database and a target database are in a data synchronization state, the data synchronization device provided by the embodiment of the present application first determines the first dynamic data table of the source database and the second dynamic data table of the target database. Subsequently, based on the data synchronization identifier, the write operation data to be synchronized that has not yet been synchronized to the second dynamic data table is identified from the first dynamic data table, and data synchronization is performed only on the write operation data to be synchronized. As a result, the data synchronization device provided by the embodiment of the present application can not only selectively synchronize only the write operation data to be synchronized, avoiding unnecessary data transmission, but also avoids the computationally intensive task of performing deduplication processing, thereby reducing the occupation of computing resources, shortening the time of the data synchronization process, and improving the efficiency of data synchronization.

[0140] The embodiments of the present application also provide corresponding data synchronization devices and computer-readable media for implementing the data synchronization method provided in the embodiments of the present application.

[0141] The data synchronization device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute instructions or codes so that the device executes a data synchronization method of any embodiment of the present application.

[0142] See also Figure 5 , which is a schematic diagram of a computer readable medium provided by an embodiment of the present application. The computer readable medium 500 stores a computer program 511, which implements the above-mentioned Figure 1 Steps of the data synchronization method.

[0143] It should be noted that in the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in conjunction with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] It should be noted that the machine-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0147] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0148] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data synchronization method, characterized in that: The method comprises: When the source database and the target database are in a data synchronization state, determining a first dynamic data table of the source database and a second dynamic data table of the target database, wherein the first dynamic data table and the second dynamic data table include write operation data, each of the write operation data includes a data synchronization flag, and the data synchronization flag indicates whether the write operation data has completed synchronization; identifying, according to the data synchronization identifier, from the first dynamic data table the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table; The write operation data to be synchronized is synchronized to the target database.

2. The method according to claim 1, characterized in that The step of identifying, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier comprises: If the comparison mode is confirmed to be sampling comparison according to the comparison mode field value in the first dynamic data table, and the number of rows in the first dynamic data table and the number of rows in the second dynamic data table are the same, then determining the sampling step length; Determining, according to the sampling step length, sampled write operation data and a row number of the sampled write operation data from the first dynamic data table; Determine the comparison write operation data in the second dynamic data table, wherein the row number of the comparison write operation data is the same as the row number of the sampling write operation data; If the sampled write operation data and the compared write operation data are inconsistent, the sampled write operation data is identified as the write operation data to be synchronized.

3. The method according to claim 1, characterized in that The step of identifying, from the first dynamic data table, the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table according to the data synchronization identifier comprises: If the comparison mode is determined to be a full comparison according to the comparison mode field value in the first dynamic data table, the first dynamic data table and the second dynamic data table are respectively paginated to obtain a paginated first dynamic data table and a paginated second dynamic data table; Sorting the paginated first dynamic data table and the paginated second dynamic data table in ascending order of primary keys to obtain a sorted first dynamic data table and a sorted second dynamic data table; If the primary key of the Mth data in the sorted first dynamic data table is consistent with the primary key of the Mth data in the sorted second dynamic data table and the other fields of the Mth data in the sorted first dynamic data table are inconsistent with the other fields of the Mth data in the sorted second dynamic data table, then the Mth data in the sorted first dynamic data table is identified as the data to be synchronized write operation.

4. The method according to claim 3, characterized in that The method further comprises: If the primary key of the Mth data in the sorted first dynamic data table is smaller than the primary key of the Mth data in the sorted second dynamic data table and the primary key of the Mth data in the sorted first dynamic data table does not exist in the primary key set corresponding to the sorted second dynamic data table, then the Mth data in the sorted first dynamic data table is identified as the data to be synchronized write operation.

5. The method according to claim 1, characterized in that The method further comprises: When the source database and the target database are in a data synchronization state, if a data synchronization anomaly is detected, and the cause of the anomaly is a network connection failure and / or a storage device failure, the data synchronization state is terminated.

6. A data synchronization device, characterized in that: The device comprises: a determination module, an identification module and a synchronization module; The determination module is used to determine a first dynamic data table of the source database and a second dynamic data table of the target database when the source database and the target database are in a data synchronization state, wherein the first dynamic data table and the second dynamic data table include write operation data, each of the write operation data includes a data synchronization identifier, and the data synchronization identifier indicates whether the write operation data has completed synchronization; The identification module is used to identify the to-be-synchronized write operation data that has not yet been synchronized to the second dynamic data table from the first dynamic data table according to the data synchronization identifier; The synchronization module is used to synchronize the write operation data to be synchronized to the target database.

7. The device according to claim 6, characterized in that The identification module specifically includes: a first identification submodule, a second identification submodule, a third identification submodule and a fourth identification submodule; The first identification submodule is used to determine a sampling step length if the comparison mode is confirmed to be a sampling comparison according to the comparison mode field value in the first dynamic data table, and the number of rows in the first dynamic data table and the second dynamic data table are consistent; The second identification submodule is used to determine the sampled write operation data and the row number of the sampled write operation data from the first dynamic data table according to the sampling step size; The third identification submodule is used to determine the comparison write operation data in the second dynamic data table, wherein the row number of the comparison write operation data is the same as the row number of the sampling write operation data; The fourth identification submodule is configured to identify the sampled write operation data as the write operation data to be synchronized if the sampled write operation data and the compared write operation data are inconsistent.

8. The device according to claim 6, characterized in that The identification module specifically includes: a fifth identification submodule, a sixth identification submodule and a seventh identification submodule; The fifth identification submodule is used for, if the comparison mode is determined to be full comparison according to the comparison mode field value in the first dynamic data table, to perform paging processing on the first dynamic data table and the second dynamic data table respectively, to obtain the first dynamic data table and the second dynamic data table after paging; The sixth identification submodule is used to sort the paginated first dynamic data table and the paginated second dynamic data table in ascending order of primary keys to obtain a sorted first dynamic data table and a sorted second dynamic data table; The seventh identification submodule is used to identify that the Mth data in the sorted first dynamic data table is the data to be synchronized write operation if the primary key of the Mth data in the sorted first dynamic data table is consistent with the primary key of the Mth data in the sorted second dynamic data table and the other fields of the Mth data in the sorted first dynamic data table are inconsistent with the other fields of the Mth data in the sorted second dynamic data table.

9. A data synchronization device, characterized in that: The device comprises: a memory and a processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the data synchronization method as described in any one of claims 1 to 5.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the data synchronization method according to any one of claims 1 to 5 is implemented.

Images