Data synchronization method and apparatus, computer device, and storage medium

Abstract

The application relates to a data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: a server sends a first pull request to a source end database to pull first comparison data in a preset time length only when a data comparison condition is met; when the first comparison data sent by the source end database is received, a second pull request is sent to a target end database to pull second comparison data corresponding to the first comparison data; the second comparison data and the first comparison data are compared according to the received second comparison data; target difference data between the source end database and the target end database is determined based on the comparison result; and the records in the target end database are synchronized according to the target difference data, with the records in the source end database as the reference, so that a large amount of invalid intermediate to-be-verified data is prevented from being accumulated on the server when the business load is high, and the problem that the server computing power is wasted due to the verification and synchronization processing of the invalid intermediate to-be-verified data by the server is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data synchronization method, apparatus, computer device, and storage medium. Background Technology

[0002] When migrating business data from a single-machine database to a distributed database, a crucial process involves a reverse synchronization window after a successful migration. This window prevents the business from quickly reverting to the original single-machine database should the migration fail. During the synchronization of incremental data from the distributed database (source database) to the single-machine database (target database), the server periodically pulls data for comparison and verification from both databases. When the business load is high, the server accumulates a large amount of invalid intermediate verification data from the single-machine database. Processing this invalid intermediate verification data for verification and synchronization wastes the server's computing power. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a data synchronization method, apparatus, computer device, and storage medium.

[0004] Firstly, this application provides a data synchronization method, including:

[0005] If the data comparison conditions are met, a first pull request is sent to the source database, wherein the first pull request is used to request the first comparison data within a preset time period;

[0006] Upon receiving the first comparison data from the source database, a second pull request is sent to the target database, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data;

[0007] Upon receiving the second comparison data, the target difference data between the source database and the target database is determined based on the comparison result between the first comparison data and the second comparison data.

[0008] The target database is synchronized based on the target difference data.

[0009] Secondly, this application provides a data synchronization device, comprising:

[0010] The first retrieval module is used to send a first retrieval request to the source database when the data comparison conditions are met. The first retrieval request is used to request the retrieval of the first comparison data within a preset time period.

[0011] The second fetch module is used to send a second fetch request to the target database when it receives the first comparison data from the source database, wherein the second fetch request is used to request the fetching of the second comparison data corresponding to the first comparison data;

[0012] The comparison and verification module is used to determine the target difference data between the source database and the target database based on the comparison result between the first comparison data and the second comparison data when the second comparison data is received.

[0013] The synchronization module is used to synchronize the target database based on the target difference data.

[0014] Thirdly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0015] If the data comparison conditions are met, a first pull request is sent to the source database, wherein the first pull request is used to request the first comparison data within a preset time period;

[0016] Upon receiving the first comparison data from the source database, a second pull request is sent to the target database, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data;

[0017] Upon receiving the second comparison data, the target difference data between the source database and the target database is determined based on the comparison result between the first comparison data and the second comparison data.

[0018] The target database is synchronized based on the target difference data.

[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0020] If the data comparison conditions are met, a first pull request is sent to the source database, wherein the first pull request is used to request the first comparison data within a preset time period;

[0021] Upon receiving the first comparison data from the source database, a second pull request is sent to the target database, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data;

[0022] Upon receiving the second comparison data, the target difference data between the source database and the target database is determined based on the comparison result between the first comparison data and the second comparison data.

[0023] The target database is synchronized based on the target difference data.

[0024] Based on the above data synchronization method, the server only sends a first pull request to the source database to pull the first comparison data within a preset time period when the data comparison conditions are met. Upon receiving the first comparison data sent by the source database, the server sends a second pull request to the target database to pull the second comparison data corresponding to the first comparison data. The received second comparison data is compared with the first comparison data, and the target difference data between the source database and the target database is determined based on the comparison result. The records in the target database are synchronized according to the target difference data, taking the records in the source database as the standard. Since the server pulls the comparison data from the source database and the target database on demand in the above data synchronization method, it avoids the accumulation of a large amount of invalid intermediate verification data on the server when the business load is high, thereby solving the problem of wasted server computing power caused by the server performing verification and synchronization processing on invalid intermediate verification data. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a diagram illustrating the application environment of a data synchronization method in one embodiment;

[0028] Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment;

[0029] Figure 3 This is a structural block diagram of a data synchronization device in one embodiment;

[0030] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Figure 1 This is a diagram illustrating the application environment of a data synchronization method in one embodiment. (Refer to...) Figure 1 This data synchronization method is applied to a data synchronization system. The system includes a source database 110, a target database 120, and a server 130. The source database 110, target database 120, and server 130 are connected via a network. The source database 110 and target database 120 are different databases; the source database 110 has a stronger data processing capability than the target database 120. In this embodiment, the source database 110 is a distributed database, while the target database 120 is a single-machine database. The server 130 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0033] In one embodiment, Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment, with reference to... Figure 2 This provides a data synchronization method. This embodiment primarily applies this method to the aforementioned... Figure 1 Taking server 130 as an example, the data synchronization method specifically includes the following steps:

[0034] Step S210: If the data comparison conditions are met, a first pull request is sent to the source database 110, wherein the first pull request is used to request the pull of the first comparison data within a preset time period.

[0035] Specifically, when the data comparison conditions are met, server 130 sends a first pull request to source database 110. Meeting the data comparison conditions indicates that the synchronous incremental data processing pressure of server 130 is relatively low, meaning that there is no data to be compared and verified at server 130, and new data to be compared and verified can be pulled from source database 110. Therefore, server 130 sends a first pull request to source database 110. When source database 110 receives the first pull request, it queries the first comparison data within a preset time period corresponding to the first pull request and sends the first comparison data to server 130. The preset time period refers to the time between the last time server 130 pulled comparison data from source database 110 and the current time when the data comparison conditions are met. The first comparison data contains record information that has changed in source database 110 within the preset time period. The record information includes record data and record identifier (ID). Record data refers to the content corresponding to the record identifier in source database 110 at the current time. Each record information indicates one record, and each record corresponds to a unique record identifier.

[0036] Step S220: Upon receiving the first comparison data from the source database 110, a second pull request is sent to the target database 120, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data.

[0037] Specifically, when the server 130 receives the first comparison data, it generates a second pull request based on each record identifier in the first comparison data. The second pull request is used to pull the corresponding record data from the target database 120 according to each record identifier in the first comparison data. That is, the second comparison data includes the record data in the target database 120 that corresponds to each record identifier in the first comparison data.

[0038] Step S230: Upon receiving the second comparison data, determine the target difference data between the source database 110 and the target database 120 based on the comparison result between the first comparison data and the second comparison data.

[0039] Specifically, when the server 130 receives the second comparison data, it performs a comparison and verification process between the first and second comparison data. That is, it compares the record data corresponding to each record identifier in the first comparison data with the record data corresponding to the same record identifier in the second comparison data to obtain a comparison result. The comparison result is used to determine whether the record data in the target database 120 is consistent with the record data in the source database 110 that indicates the same record identifier. In other words, it determines whether the record data in the target database 120 and the source database 110 corresponding to the same record identifier are synchronized. If the comparison result indicates that the record data in the target database 120 and the source database 110 with the same record identifier are synchronized, then the target difference data is determined to be empty. If the comparison result indicates that the record data in the target database 120 and the source database 110 with the same record identifier are not synchronized, then the record identifiers that are not synchronized between the target database 120 and the source database 110 are taken as the target difference data.

[0040] Step S240: Synchronize the target database 120 according to the target difference data.

[0041] Specifically, the target database 120 is synchronized using the target difference data, ensuring that the records in the target database 120 are consistent with the records corresponding to the same record identifier in the source database 110. That is, the records in the source database 110 are used as the standard to synchronize the records in the target database 120 according to the target difference data. Since the server 130 automatically pulls comparison data from the source database 110 and the target database 120 as needed in the above data synchronization method, it avoids the accumulation of a large amount of invalid intermediate verification data on the server 130 when the business load is high. This solves the problem of wasted computing power of the server 130 due to the verification and synchronization of invalid intermediate verification data.

[0042] In one embodiment, sending a first fetch request to the source database 110 when the data comparison conditions are met includes:

[0043] Upon reaching the preset comparison time, a first fetch request is sent to the source database 110; or,

[0044] Upon receiving the signal that the previous batch of data has been synchronized, a first fetch request is sent to the source database 110.

[0045] Specifically, the data comparison conditions include a preset comparison time and a signal indicating the completion of the previous batch of data synchronization. The preset comparison time refers to the time when the server 130 is powered on and running, at which point the first synchronization verification process begins. Therefore, when the preset comparison time is reached, the server 130 sends its first pull request to the source database 110 to initiate the first batch of data synchronization. Subsequently, each time the server 130 receives a signal indicating the completion of the previous batch of data synchronization, it signifies that the server 130 has completed a batch of data synchronization processing. This also means that there is no backlog of data to be compared and verified at the server 130, and it can continue to pull new data to be compared from the source database 110 to initiate a new batch of data synchronization processing.

[0046] In one embodiment, sending a second fetch request to the target database 120 upon receiving the first comparison data from the source database 110 includes:

[0047] Upon receiving the first comparison data from the source database 110, the first comparison data is split into multiple comparison tasks, wherein the first comparison data includes multiple first records to be compared;

[0048] The comparison tasks that have not been compared are selected sequentially by sliding the preset sliding window and saved as target tasks to the task queue. The number of target tasks in the task queue is equal to the number of windows in the preset sliding window.

[0049] A comparison thread is started for each target task in the task queue;

[0050] Each comparison thread generates a second pull request corresponding to each target task based on the record identifier of each first comparison record in the corresponding target task. The second pull request is used to request the retrieval of a second comparison record corresponding to the record identifier of each first comparison record in the corresponding target task from the target database 120. The second comparison data includes multiple second comparison records.

[0051] Each of the second pull requests is sent to the target database 120.

[0052] Specifically, upon receiving the first comparison data sent from the source database 110, the first comparison data is split according to a preset number of records. That is, multiple first comparison records in the first comparison data are split into multiple comparison tasks. In other words, the number of first comparison records in each comparison task is equal to the preset number of records. For example, if the preset number of records is 1000, then every 1000 first comparison records in the first comparison data will form one comparison task. And the comparison tasks are arranged in chronological order.

[0053] The preset sliding window contains multiple windows, each window corresponding to a comparison thread. By sliding the preset sliding window, uncompared comparison tasks are selected sequentially as target tasks. The target tasks selected by the preset sliding window are saved to the task queue. The target tasks selected by each window in the preset sliding window are compared and verified by the comparison thread corresponding to that window. Each comparison thread generates a second pull request corresponding to each target task based on the record identifier of each first record to be compared in the received target task, and sends the second pull request to the target database 120. The second record to be compared is retrieved from the target database 120 according to the record identifier of each first record to be compared. That is, the second record to be compared is searched in the target database 120 according to the record identifier of the first record to be compared.

[0054] In one embodiment, determining the target difference data between the source database 110 and the target database 120 based on the comparison result between the first comparison data and the second comparison data includes:

[0055] Each comparison thread performs a comparison process with each first record to be compared and multiple second records to be compared in each target task according to a preset comparison scheme, thereby obtaining the comparison results corresponding to each target task;

[0056] Determine the comparison status of each of the comparison tasks;

[0057] If there are comparison tasks whose comparison status is not compared, the step of selecting the uncompared comparison tasks as target tasks and saving them to the task queue by sliding the preset sliding window in sequence until the comparison results corresponding to each target task are obtained is executed.

[0058] If all the comparison tasks are in the comparison status of "compared", the target difference data between the source database 110 and the target database 120 is determined based on the comparison results corresponding to all the comparison tasks.

[0059] Specifically, each comparison thread iterates and compares each first comparison record in the corresponding target task with multiple second comparison records according to a preset comparison scheme, thereby verifying whether there is a second comparison record that matches the first comparison record, thus determining the comparison result of the first comparison record, and determining the comparison result corresponding to the target task based on the comparison results of all first comparison records in the target task.

[0060] The comparison status of each comparison task is determined, including both uncompared and compared. This involves checking if any comparison tasks remain unselected by the preset sliding window. If uncompared tasks exist, the preset sliding window continues to slide to select them as target tasks and save them to the task queue until the comparison results for each target task are obtained. When at least one comparison thread within the preset sliding window obtains a comparison result for its corresponding target task, the preset sliding window is then slid to select uncompared tasks. The sliding step size of the preset sliding window is the same as the number of target task comparison results obtained within the window. For example, if one window's comparison result is obtained, the sliding step size is one window, used to select the next uncompared task; if two windows' comparison results are obtained, the sliding step size is two windows, used to select the two uncompared tasks. The amount of data that the server 130 compares and verifies is limited by a preset sliding window, and comparisons and verifications are only performed on target tasks located within the preset sliding window.

[0061] If all comparison tasks are in the "comparison completed" state, there is no need to continue sliding the preset sliding window. The target difference data between the source database 110 and the target database 120 is determined based on the comparison results of all comparison tasks.

[0062] In one embodiment, the step of comparing each first comparison record in each target task with multiple second comparison records in each target task according to a preset comparison scheme by each comparison thread to obtain the comparison result corresponding to each target task includes:

[0063] The first matching result between the record identifier of the first record to be compared in the current target task and the record identifiers of each of the second records to be compared is determined by the current comparison thread, wherein the current first record to be compared is any uncompared first record to be compared in the current target task;

[0064] If at least one of the first matching results is a successful match, the second record to be compared with which the matching result is a successful match is taken as a candidate record, and a second matching result is determined between the time field of the candidate record and the time field of the current first matching record;

[0065] If the second matching result is a successful match, a third matching result is determined between the cyclic redundancy check value corresponding to the candidate record and the cyclic redundancy check value corresponding to the current first comparison record.

[0066] Based on the first matching result, second matching result, or third matching result corresponding to each of the first records to be compared, the comparison result corresponding to the current target task is determined.

[0067] Specifically, the current comparison thread is any one of the comparison threads corresponding to the preset sliding window, and the current target task is the target task corresponding to the current comparison thread. The current comparison thread compares and verifies each first record to be compared with each second record to be compared in the current target task. The current first record to be compared is any one of the first records to be compared in the current target task. The current comparison thread first compares and verifies the record identifier of the current first record to be compared with the record identifier of each second record to be compared, thereby obtaining the first matching result between the current first record to be compared and each second record to be compared. The first matching result includes successful matching and failed matching. A successful first matching result means that the record identifier of the current first record to be compared is the same as the record identifier of the second record to be compared, that is, the current first record to be compared and the second record to be compared indicate the same record. A failed first matching result means that the record identifier of the current first record to be compared is different from the record identifier of the second record to be compared, that is, the current first record to be compared and the second record to be compared indicate different records.

[0068] If at least one first matching result is successful, it means that there is a second record in the target database 120 with the same record identifier as the current first record to be compared. The second record with the same record identifier as the current first record to be compared is then taken as a candidate record. The time field of the current first record to be compared is then checked against the time field of the candidate record. The time field is used to indicate the time of the record's creation. If the time field of the current first record to be compared is the same as the time field of the candidate record, a second matching result is obtained, indicating that the candidate record and the current first record to be compared were created at the same time. The cyclic redundancy check value (CRC check value) corresponding to the candidate record is then checked against the cyclic redundancy check value corresponding to the current first record to be compared, resulting in a third matching result. If the third matching result is successful, it means that the candidate record and the current first record to be compared are the same record, that is, the candidate record in the target database 120 and the current first record to be compared in the source database 110 have been synchronized and there is no difference in data.

[0069] Similarly, the current comparison thread performs comparison processing on each first record to be compared in the current target task, thereby obtaining the matching result corresponding to each first record to be compared. The matching result is the first matching result, the second matching result, or the third matching result. If the matching result corresponding to each first record to be compared is successful, the comparison result corresponding to the current target task is determined to be successful; if the matching result corresponding to at least one first record to be compared is unsuccessful, the comparison result corresponding to the current target task is determined to be unsuccessful.

[0070] In one embodiment, after determining the first matching result between the record identifier of the first record to be compared and the record identifiers of each of the second records to be compared in the current target task through the current comparison thread, the method further includes:

[0071] If the first matching result is a failure, the record identifier of the current first record to be compared is saved to the difference file; or,

[0072] If the second matching result is a match failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database 120; or,

[0073] If the third matching result is a failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database 120.

[0074] Specifically, if the first matching result corresponding to the current first record to be compared is a failed match, it means that the current first record to be compared was not found in the target database 120. In this case, the current first record to be compared is treated as a difference record, and the record identifier of the difference record is saved to the difference file. That is, the difference file contains the record identifiers of the difference records between the target database 120 and the source database 110. If the first matching result corresponding to the current first record to be compared is a successful match, but the corresponding second matching result is a failed match, it means that a second record to be compared (candidate record) corresponding to the current first record to be compared is found in the target database 120. However, the candidate record and the current first record to be compared are records generated at different times. In this case, the current first record to be compared is treated as a difference record, and the record identifier of the difference record is saved to the difference file. A deletion command is also generated based on the record identifier of the difference record and sent to the target database 120 to delete the difference record in the target database.

[0075] If the first matching result for the current first record to be compared is a successful match, and the second matching result is a successful match, but the corresponding third matching result is a failed match, it indicates that the candidate record is not the current first record to be compared. In this case, the candidate record is treated as a difference record, its record identifier is saved to a difference file, and a deletion command is generated based on the record identifier and sent to the target database 120 to delete the difference record from the target database. In this embodiment, only the current first record to be compared is used as an example. Each first record to be compared in the first comparison data undergoes the comparison and verification process described above, which will not be repeated here.

[0076] Furthermore, when the server 130 receives the first comparison data, it also performs deduplication processing on the difference file based on the first comparison data, that is, it removes record identifiers in the difference file that are the same as the record identifiers of the first record to be compared in the first comparison data. The server 130 performs difference synchronization processing in each batch using the record identifiers in the difference file.

[0077] In one embodiment, the target difference data includes the identifiers of each record in the difference file, and the synchronization processing of the target database 120 based on the target difference data includes:

[0078] A third fetch request is generated based on the target difference data, and the third fetch request is sent to the source database 110. The third fetch request is used to request the fetching of the update record corresponding to the target difference data from the source database 110.

[0079] Upon receiving an update record corresponding to the target difference data from the source database 110, a clearing instruction is generated based on the record identifier of the update record, wherein the clearing instruction is used to clear the record corresponding to the record identifier of the update record in the target database 120;

[0080] Upon receiving the completion signal corresponding to the clearing instruction, a corresponding synchronization instruction is generated based on the update record, and the synchronization instruction is sent to the target database 120. The synchronization instruction is used to insert the update record into the target database 120 to perform synchronization processing on the target database 120.

[0081] Specifically, the target difference data is the difference file. A third fetch request is generated based on the target difference data and sent to the source database 110 to fetch the update records corresponding to the target difference data. Update records refer to records in the target difference data that match the record identifier of the difference record and whose generation time is within a preset time period. The server 130 generates a clear instruction (delete statement script) based on the update records from the source database 110 and sends it to the target database 120 to clear the records corresponding to the record identifiers of the update records in the target database 120. After clearing, a synchronization instruction (insert statement script) is sent to the target database 120 to insert the update records into the target database 120, ensuring that the records in the target database 120 are consistent with the records in the source database 110. Thus, once the synchronization verification process for one batch is completed, a data synchronization completion signal is generated. Afterwards, the server 130 continues to execute the step of sending a first pull request to the source database 110 upon receiving the previous batch data synchronization completion signal, initiating a new batch data synchronization process. That is, the server 130 flexibly pulls data to be compared from both the source database 110 and the target database within the same time period according to its own data synchronization progress for comparison verification. Compared to the existing technology of periodically sending data to be compared to the server 130, this method reduces the amount of synchronization verification data for the server 130 and avoids the accumulation of a large amount of invalid intermediate data to be verified on the server 130 when the business load is high. This solves the problem of wasted computing power on the server 130 due to the server performing verification and synchronization processing on invalid intermediate data.

[0082] Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment. It should be understood that, although... Figure 2The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0083] In one embodiment, such as Figure 3 As shown, a data synchronization device is provided, comprising:

[0084] The first retrieval module 310 is used to send a first retrieval request to the source database 110 when the data comparison conditions are met. The first retrieval request is used to request the retrieval of the first comparison data within a preset time period.

[0085] The second pull module 320 is used to send a second pull request to the target database 120 when it receives the first comparison data from the source database 110, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data;

[0086] The comparison and verification module 330 is used to determine the target difference data between the source database 110 and the target database 120 based on the comparison result between the first comparison data and the second comparison data when the second comparison data is received.

[0087] The synchronization module 340 is used to perform synchronization processing on the target database 120 based on the target difference data.

[0088] In one embodiment, the first pull module 310 is further configured to:

[0089] Upon reaching the preset comparison time, a first fetch request is sent to the source database 110; or,

[0090] Upon receiving the signal that the previous batch of data has been synchronized, a first fetch request is sent to the source database 110.

[0091] In one embodiment, the second pull module 320 is further configured to:

[0092] Upon receiving the first comparison data from the source database 110, the first comparison data is split into multiple comparison tasks, wherein the first comparison data includes multiple first records to be compared;

[0093] The comparison tasks that have not been compared are selected sequentially by sliding the preset sliding window and saved as target tasks to the task queue. The number of target tasks in the task queue is equal to the number of windows in the preset sliding window.

[0094] A comparison thread is started for each target task in the task queue;

[0095] Each comparison thread generates a second pull request corresponding to each target task based on the record identifier of each first comparison record in the corresponding target task. The second pull request is used to request the retrieval of a second comparison record corresponding to the record identifier of each first comparison record in the corresponding target task from the target database 120. The second comparison data includes multiple second comparison records.

[0096] Each of the second pull requests is sent to the target database 120.

[0097] In one embodiment, the comparison and verification module 330 is further configured to:

[0098] Each comparison thread performs a comparison process with each first record to be compared and multiple second records to be compared in each target task according to a preset comparison scheme, thereby obtaining the comparison results corresponding to each target task;

[0099] Determine the comparison status of each of the comparison tasks;

[0100] If there are comparison tasks whose comparison status is not compared, the step of selecting the uncompared comparison tasks as target tasks and saving them to the task queue by sliding the preset sliding window in sequence until the comparison results corresponding to each target task are obtained is executed.

[0101] If all the comparison tasks are in the comparison status of "compared", the target difference data between the source database 110 and the target database 120 is determined based on the comparison results corresponding to all the comparison tasks.

[0102] In one embodiment, the comparison and verification module 330 is further configured to:

[0103] The first matching result between the record identifier of the first record to be compared in the current target task and the record identifiers of each of the second records to be compared is determined by the current comparison thread, wherein the current first record to be compared is any uncompared first record to be compared in the current target task;

[0104] If at least one of the first matching results is a successful match, the second record to be compared with which the matching result is a successful match is taken as a candidate record, and a second matching result is determined between the time field of the candidate record and the time field of the current first matching record;

[0105] If the second matching result is a successful match, a third matching result is determined between the cyclic redundancy check value corresponding to the candidate record and the cyclic redundancy check value corresponding to the current first comparison record.

[0106] Based on the first matching result, second matching result, or third matching result corresponding to each of the first records to be compared, the comparison result corresponding to the current target task is determined.

[0107] In one embodiment, the comparison and verification module 330 is further configured to:

[0108] If the first matching result is a failure, the record identifier of the current first record to be compared is saved to the difference file; or,

[0109] If the second matching result is a match failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database 120; or,

[0110] If the third matching result is a failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database 120.

[0111] In one embodiment, the synchronization module 340 is further configured to:

[0112] A third fetch request is generated based on the target difference data, and the third fetch request is sent to the source database 110. The third fetch request is used to request the fetching of the update record corresponding to the target difference data from the source database 110.

[0113] Upon receiving an update record corresponding to the target difference data from the source database 110, a clearing instruction is generated based on the record identifier of the update record, wherein the clearing instruction is used to clear the record corresponding to the record identifier of the update record in the target database 120;

[0114] Upon receiving the completion signal corresponding to the clearing instruction, a corresponding synchronization instruction is generated based on the update record, and the synchronization instruction is sent to the target database 120. The synchronization instruction is used to insert the update record into the target database 120 to perform synchronization processing on the target database 120.

[0115] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. Specifically, this computer device may be... Figure 1 Server 130 in the middle. For example... Figure 4 As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and may also store computer programs. When executed by the processor, these computer programs enable the processor to implement data synchronization methods. The internal memory may also store computer programs, which, when executed by the processor, enable the processor to implement data synchronization methods. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0116] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one embodiment, the data synchronization device provided in this application can be implemented as a computer program, which can be implemented in the form of, for example, Figure 4 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the data synchronization device, for example, Figure 3 The diagram shows a first fetch module 310, a second fetch module 320, a comparison and verification module 330, and a synchronization module 340. The computer program comprised of these modules causes the processor to execute the steps of the data synchronization methods in the various embodiments of this application described in this specification.

[0118] Figure 4 The computer equipment shown can be used as follows Figure 3 The data synchronization device shown includes a first fetch module 310 that, upon meeting data comparison conditions, sends a first fetch request to the source database 110. This first fetch request requests first comparison data within a preset time period. The computer device, via a second fetch module 320, upon receiving the first comparison data from the source database 110, sends a second fetch request to the target database 120. This second fetch request requests second comparison data corresponding to the first comparison data. The computer device, via a comparison verification module 330, upon receiving the second comparison data, determines target difference data between the source database 110 and the target database 120 based on the comparison result between the first and second comparison data. The computer device, via a synchronization module 340, performs synchronization processing on the target database 120 based on the target difference data.

[0119] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0120] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data synchronization method, characterized in that, The method includes: If the data comparison conditions are met, a first pull request is sent to the source database, wherein the first pull request is used to request the first comparison data within a preset time period; Upon receiving the first comparison data from the source database, a second pull request is sent to the target database, wherein the second pull request is used to request the pull of the second comparison data corresponding to the first comparison data; Upon receiving the second comparison data, the target difference data between the source database and the target database is determined based on the comparison result between the first comparison data and the second comparison data. This includes: using each comparison thread to compare each first comparison record in each target task with multiple second comparison records according to a preset comparison scheme, obtaining the comparison result corresponding to each target task; determining the comparison status of each comparison task; if there are comparison tasks with a comparison status of "not compared", the uncompared comparison tasks are selected sequentially by sliding a preset sliding window as target tasks and saved to the task queue until the comparison result corresponding to each target task is obtained; if the comparison status of all comparison tasks is "compared", the target difference data between the source database and the target database is determined based on the comparison result corresponding to all comparison tasks. The target database is synchronized based on the target difference data. The process involves each comparison thread performing a comparison with multiple second comparison records in each target task according to a preset comparison scheme, thereby obtaining the comparison results corresponding to each target task. This includes: determining a first matching result between the record identifier of the current first comparison record and the record identifiers of each second comparison record in the current target task, where the current first comparison record is any uncompared first comparison record in the current target task; if at least one of the first matching results is successful, using the second comparison record with the successful matching result as a candidate record, and determining a second matching result between the time field of the candidate record and the time field of the current first comparison record; if the second matching result is successful, determining a third matching result between the cyclic redundancy check value corresponding to the candidate record and the cyclic redundancy check value corresponding to the current first comparison record; and determining the comparison result corresponding to the current target task based on the first matching result, second matching result, or third matching result corresponding to each first comparison record.

2. The method according to claim 1, characterized in that, The step of sending a first fetch request to the source database when the data comparison conditions are met includes: Upon reaching the preset comparison time, a first fetch request is sent to the source database; or, Upon receiving the signal that the previous batch of data has been synchronized, a first pull request is sent to the source database.

3. The method according to claim 1, characterized in that, Upon receiving the first comparison data from the source database, sending a second fetch request to the target database includes: Upon receiving the first comparison data from the source database, the first comparison data is split into multiple comparison tasks, wherein the first comparison data includes multiple first records to be compared; The comparison tasks that have not been compared are selected sequentially by sliding the preset sliding window and saved as target tasks to the task queue. The number of target tasks in the task queue is equal to the number of windows in the preset sliding window. A comparison thread is started for each target task in the task queue; Each comparison thread generates a second fetch request corresponding to each target task based on the record identifier of each first comparison record in the corresponding target task. The second fetch request is used to request the fetching of a second comparison record corresponding to the record identifier of each first comparison record in the corresponding target task from the target database. The second comparison data includes multiple second comparison records. Each of the second pull requests is sent to the target database.

4. The method according to claim 1, characterized in that, After determining the first matching result between the record identifier of the first record to be compared and the record identifiers of each of the second records to be compared in the current target task through the current comparison thread, the method further includes: If the first matching result is a failure, the record identifier of the current first record to be compared is saved to the difference file; or, If the second matching result is a match failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database; or, If the third matching result is a match failure, the record identifier of the current first record to be compared is saved to the difference file, and a corresponding deletion instruction is generated based on the candidate record and sent to the target database.

5. The method according to claim 4, characterized in that, The target difference data includes the identifiers of each record in the difference file, and the synchronization process of the target database based on the target difference data includes: A third fetch request is generated based on the target difference data, and the third fetch request is sent to the source database. The third fetch request is used to request the fetching of the update record corresponding to the target difference data from the source database. Upon receiving an update record corresponding to the target difference data from the source database, a clearing instruction is generated based on the record identifier of the update record, wherein the clearing instruction is used to clear the record corresponding to the record identifier of the update record in the target database; Upon receiving the completion signal corresponding to the clearing instruction, a corresponding synchronization instruction is generated based on the update record, and the synchronization instruction is sent to the target database. The synchronization instruction is used to insert the update record into the target database to perform synchronization processing on the target database.

6. A data synchronization device, characterized in that, The device includes: The first retrieval module is used to send a first retrieval request to the source database when the data comparison conditions are met. The first retrieval request is used to request the retrieval of the first comparison data within a preset time period. The second fetch module is used to send a second fetch request to the target database when it receives the first comparison data from the source database, wherein the second fetch request is used to request the fetching of the second comparison data corresponding to the first comparison data; The comparison and verification module is used to determine the target difference data between the source database and the target database based on the comparison result between the first comparison data and the second comparison data when the second comparison data is received. This includes: performing comparison processing on each first comparison record in each target task with multiple second comparison records according to a preset comparison scheme using each comparison thread, to obtain the comparison result corresponding to each target task; determining the comparison status of each comparison task; if there are comparison tasks with a comparison status of "not compared", executing the step of sequentially selecting the uncompared comparison tasks as target tasks and saving them to the task queue according to a preset sliding window until the comparison result corresponding to each target task is obtained; and if the comparison status of all comparison tasks is "already compared", determining the target difference data between the source database and the target database based on the comparison results corresponding to all comparison tasks. The synchronization module is used to synchronize the target database based on the target difference data. The process involves each comparison thread performing a comparison with multiple second comparison records in each target task according to a preset comparison scheme, thereby obtaining the comparison results corresponding to each target task. This includes: determining a first matching result between the record identifier of the current first comparison record and the record identifiers of each second comparison record in the current target task, where the current first comparison record is any uncompared first comparison record in the current target task; if at least one of the first matching results is successful, using the second comparison record with the successful matching result as a candidate record, and determining a second matching result between the time field of the candidate record and the time field of the current first comparison record; if the second matching result is successful, determining a third matching result between the cyclic redundancy check value corresponding to the candidate record and the cyclic redundancy check value corresponding to the current first comparison record; and determining the comparison result corresponding to the current target task based on the first matching result, second matching result, or third matching result corresponding to each first comparison record.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images