A data synchronization method, storage medium and device

Abstract

This invention provides a data synchronization method, storage medium, and device. The data synchronization method includes: obtaining one incremental data entry from each source database; sending the obtained incremental data to a coordination node, wherein the coordination node is located between a target database and each of the source databases and is used to establish a synchronization link between the target database and each of the source databases; determining the incremental data with the highest priority among the coordination nodes; and delivering the incremental data with the highest priority to the target database. The advantage of this invention is that it can ensure the accuracy of data synchronization and accelerate the data synchronization speed in many-to-one data synchronization scenarios.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a data synchronization method, storage medium, and device. Background Technology

[0002] Real-time data synchronization using data synchronization tools generally consists of three stages. The first stage involves initializing and loading existing data to establish a baseline for data synchronization. The second stage uses the established baseline as a benchmark for incremental data synchronization. The third stage involves periodically comparing and verifying the source and target data during data synchronization to ensure that no data is lost during the synchronization process.

[0003] One type of data synchronization use case is many-to-one data synchronization. For example, synchronizing data from source databases A and B to a target database C. The current approach involves deploying separate synchronization links between the target database C and the source databases A and B. These two links operate independently, synchronizing data from source databases A and B to the target database C separately.

[0004] However, in practical applications, the order in which data is synchronized between source databases A and B to the target database C may be critical. For example, it might be necessary to insert one record from source database A into the target database C first, followed by one record from source database B. An incorrect order could lead to synchronization failure, thus requiring further improvement. Summary of the Invention

[0005] One object of the present invention is to provide a data synchronization method, storage medium, and device that overcomes or at least partially solves the above-mentioned problems.

[0006] A further objective of this invention is to reduce the processing logic complexity of the coordinating node.

[0007] Specifically, the present invention provides a data synchronization method, comprising:

[0008] Retrieve one incremental data entry from each source database;

[0009] The acquired incremental data is sent to the coordination node, which is located between the target database and each of the source databases and is used to associate the synchronization link between the target database and each of the source databases.

[0010] Identify the highest priority incremental data among the coordinating nodes;

[0011] The highest priority incremental data is delivered to the target database.

[0012] Optionally, the step of retrieving one incremental data record from each source database includes:

[0013] Parse the log tables of each source database;

[0014] Retrieve incremental data with global tags from the parsed log table.

[0015] Optionally, the global marker is a timestamp, and the step of determining the highest priority incremental data in the coordination node includes:

[0016] Compare the timestamps of the incremental data in the coordination node;

[0017] The incremental data with the smallest timestamp is selected as the highest priority incremental data.

[0018] Optionally, the global marker is a change number or a sequence number, and the step of determining the highest priority incremental data in the coordinating node includes:

[0019] Convert the change number or sequence number of each incremental data in the coordination node into a timestamp.

[0020] Compare the timestamps of the incremental data in the coordination node;

[0021] The incremental data with the smallest timestamp is selected as the highest priority incremental data.

[0022] Optionally, after the step of delivering the highest priority incremental data to the target database, the method further includes:

[0023] Determine whether the incremental data still exists in the source database where the incremental data has been delivered;

[0024] If so, retrieve the next incremental data from the source database where incremental data has already been delivered;

[0025] The acquired incremental data is sent to the coordinating node.

[0026] Optionally, after determining whether the incremental data still exists in the source database where the incremental data has been delivered, the method further includes:

[0027] If no incremental data exists, a virtual data entry is returned to the coordinating node, and the priority of the virtual data entry is lower than the priority of the incremental data already existing in the coordinating node.

[0028] Optionally, the timestamp of the virtual data is set to the timestamp when it is returned to the coordinating node.

[0029] Optionally, the step of delivering the highest priority incremental data to the target database includes:

[0030] The highest priority incremental data is sent to the execution node, which is located between the coordinating node and the target database.

[0031] The highest priority incremental data is processed and then delivered to the target database.

[0032] According to another aspect of the present invention, a machine-readable storage medium is also provided having a machine-executable program stored thereon, which, when executed by a processor, implements the data synchronization method described in any one of the above descriptions.

[0033] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor executes the machine-executable program to implement the data synchronization method described in any one of the above descriptions.

[0034] The data synchronization method of this invention first retrieves one incremental data entry from each source database, then sends the retrieved incremental data to a coordinating node, determines the highest-priority incremental data in the coordinating node, and finally delivers the highest-priority incremental data to the target database. In other words, this invention introduces a coordinating node into the many-to-one synchronization link, connecting the originally independent synchronization links simultaneously to the coordinating node. By determining the highest-priority incremental data at the coordinating node and prioritizing its delivery to the target database, the accuracy of data synchronization can be guaranteed, and data synchronization failure can be avoided.

[0035] Furthermore, in the data synchronization method of the present invention, an execution node is set up between the coordinating node and the target database. During the process of delivering the highest-priority incremental data to the target database, the highest-priority incremental data can be sent to the execution node first. The execution node then processes the incremental data and delivers the processed incremental data to the target database. This significantly reduces the processing logic complexity of the coordinating node, increases data throughput, and accelerates data synchronization.

[0036] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0038] Figure 1 This is a schematic diagram of the architecture of the many-to-one data synchronization method in the existing technology;

[0039] Figure 2 This is a schematic architecture diagram of a data synchronization method according to an embodiment of the present invention;

[0040] Figure 3 This is a flowchart of a data synchronization method according to an embodiment of the present invention;

[0041] Figure 4 This is a schematic flowchart illustrating the data delivery process according to an embodiment of the present invention;

[0042] Figure 5 This is a schematic architecture diagram of a data synchronization method according to another embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of a machine-readable storage medium according to an embodiment of the present invention;

[0044] Figure 7 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0045] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0046] Figure 1 This is a schematic diagram of the architecture of a many-to-one data synchronization method in existing technologies, for reference. Figure 1In many-to-one data synchronization scenarios, such as synchronizing data from source databases A and B to a target database C, the current approach involves deploying separate synchronization links between the target database C and the source databases A and B. These two links operate independently, synchronizing data from source databases A and B to the target database C separately. As mentioned earlier, the order in which data from source databases A and B is synchronized to the target database C may be critical. For example, it might be necessary to insert one record from source database A into the target database C first, followed by one record from source database B. Incorrect order can lead to synchronization failure.

[0047] Figure 2 This is a schematic architecture diagram of a data synchronization method according to an embodiment of the present invention, with reference to... Figure 2 To address the aforementioned issues, this invention introduces a coordinating node into the many-to-one synchronization link, and performs data synchronization based on this coordinating node to ensure the accuracy of data synchronization.

[0048] Figure 3 This is a flowchart of a data synchronization method according to an embodiment of the present invention, with reference to... Figure 3 The data synchronization method of the present invention includes at least the following steps S302 to S308.

[0049] Step S302: Obtain one incremental data entry from each source database.

[0050] Step S304: The acquired incremental data is sent to the coordination node. The coordination node is located between the target database and each source database and is used to associate the synchronization link between the target database and each source database.

[0051] Step S306: Determine the highest priority incremental data among the coordinating nodes.

[0052] Step S308: Deliver the highest priority incremental data to the target database.

[0053] The data synchronization method of this invention first retrieves one incremental data entry from each source database, then sends the retrieved incremental data to the coordinating node, determines the highest-priority incremental data in the coordinating node, and finally delivers the highest-priority incremental data to the target database. In other words, this invention introduces a coordinating node into the many-to-one synchronization link, connecting the originally independent synchronization links simultaneously to the coordinating node. By determining the highest-priority incremental data at the coordinating node and prioritizing its delivery to the target database, the accuracy of data synchronization can be guaranteed, and data synchronization failure can be avoided.

[0054] It is understandable that the log table of the source database records the addition, deletion and change of data. In step S102 above, the step of obtaining one incremental data from each source database can be to first parse the log table of each source database, and then obtain the incremental data with global marker from the parsed log table.

[0055] Generally, when the source database records incremental data in its log table, it usually includes a global flag. This global flag can be the current system timestamp, or it can be a database change number or sequence number. The log table also provides database functions to convert change numbers or sequence numbers into timestamps. The names of these global flags differ across databases; for example, Oracle uses SCN (System Change Number), and SQL Server uses LSN (LogSequence Number), and so on.

[0056] In an optional embodiment of the present invention, the global marker can be a timestamp. The step of determining the highest priority incremental data in the coordination node can be to compare the timestamp sizes of each incremental data in the coordination node, and then take the incremental data with the smallest timestamp as the highest priority incremental data. That is, when the global marker is a timestamp, the highest priority incremental data can be directly determined by comparing the timestamp sizes.

[0057] In another optional embodiment of the present invention, the global marker can be a change number or a sequence number. The step of determining the highest priority incremental data in the coordination node can be to first convert the change number or sequence number of each incremental data in the coordination node into a timestamp, then compare the timestamps of each incremental data in the coordination node, and finally select the incremental data with the smallest timestamp as the highest priority incremental data. That is, when the global marker is a change number or a sequence number, a database function can be used to convert the change number or sequence number into a timestamp, making the determination easier by comparing timestamps.

[0058] Figure 4 This is a schematic flowchart illustrating the data delivery process according to an embodiment of the present invention, with reference to... Figure 4 After delivering the highest priority incremental data to the target database, the process may further include steps S402 to S408. It may also include:

[0059] Step S402: Determine whether the incremental data still exists in the source database of the delivered incremental data. If yes, proceed to step S404; otherwise, proceed to step S406.

[0060] Step S404: Obtain the next incremental data from the source database where incremental data has already been delivered, and then return to step S304, that is: send the obtained incremental data to the coordinating node.

[0061] Step S406: Return a virtual data to the coordinating node, and the priority of the virtual data is lower than the priority of the incremental data that already exists in the coordinating node. Then return to step S306, that is: determine the incremental data with the highest priority in the coordinating node.

[0062] It's understandable that if incremental data still exists in the source database where incremental data has already been delivered, then the next piece of incremental data is retrieved and sent to the coordinating node. The highest-priority incremental data in the coordinating node is then re-determined. If no incremental data exists in the source database where incremental data has already been delivered, a dummy data record can be returned to the coordinating node. This dummy data record has a lower priority than other incremental data records in the coordinating node, providing a reference for the synchronization of other incremental data records and thus not blocking the data synchronization of other source databases.

[0063] In this embodiment of the invention, the timestamp of the virtual data is set to the timestamp when it is returned to the coordinating node. That is, when returning virtual data to the coordinating node, the current system event can be used as the timestamp of the virtual data. This method can more easily ensure that the priority of the virtual data is lower than the priority of any incremental data that already exists in the coordinating node.

[0064] Figure 5 This is a schematic architecture diagram of a data synchronization method according to another embodiment of the present invention, with reference to... Figure 5 An execution node is also set up between the coordinating node and the target database. In step S308 above, the step of delivering the highest priority incremental data to the target database can be: first, sending the highest priority incremental data to the execution node; then processing the highest priority incremental data; and finally, delivering the processed data to the target database. Since the introduction of the coordinating node may become a bottleneck in the entire system operation, this invention makes the coordinating node only responsible for comparing the timestamps of the incremental data, while the processing of the incremental data is completed by the execution node. This significantly reduces the processing logic complexity of the coordinating node, increases data throughput, and accelerates data synchronization.

[0065] This embodiment also provides a machine-readable storage medium 10 and a computer device 20. Figure 6 This is a schematic diagram of a machine-readable storage medium 10 according to an embodiment of the present invention. Figure 7 This is a schematic diagram of a computer device 20 according to an embodiment of the present invention.

[0066] The machine-readable storage medium 10 stores a machine-executable program 11 thereon, which, when executed by a processor, implements the data synchronization method of any of the above embodiments.

[0067] The computer device 20 may include a memory 210, a processor 220, and a machine-executable program 11 stored on the memory 210 and running on the processor 220, and the processor 220 implements the data synchronization method of any of the above embodiments when executing the machine-executable program 11.

[0068] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-based system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0069] For the purposes of this embodiment, the machine-readable storage medium 10 can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the machine-readable storage medium 10 can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0070] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0071] Computer device 20 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 20 can be a cloud computing node. Computer device 20 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 50 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0072] Computer device 20 may include a processor 220 adapted to execute stored instructions and a memory 210 that provides temporary storage space for the operation of said instructions during operation. Processor 220 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. Memory 210 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0073] The processor 220 can be connected via a system interconnect (e.g., PCI, PCI-Express, etc.) to an I / O interface (input / output interface) suitable for connecting the computer device 20 to one or more I / O devices (input / output devices). I / O devices may include, for example, a keyboard and indicating devices, where indicating devices may include a touchpad or touchscreen, etc. I / O devices may be built into the computer device 20 or may be external devices connected to the computing device.

[0074] The processor 220 may also be linked via a system interconnect to a display interface suitable for connecting the computer device 20 to a display device. The display device may include a display screen that is a built-in component of the computer device 20. The display device may also include an external computer monitor, television, or projector connected to the computer device 20. Furthermore, a network interface controller (NIC) may be adapted to connect the computer device 20 to a network via a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an Internet Minicomputer System Interface) to transmit data. The network may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. Remote devices may connect to the computing device via the network.

[0075] The flowcharts provided in this embodiment are not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0076] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data synchronization method, comprising: Retrieve one incremental data entry from each source database; The acquired incremental data is sent to the coordination node, which is located between the target database and each of the source databases and is used to associate the synchronization link between the target database and each of the source databases. Identify the highest priority incremental data among the coordinating nodes; The highest priority incremental data is delivered to the target database; Determine whether the incremental data still exists in the source database where the incremental data has been delivered; If so, retrieve the next incremental data from the source database where incremental data has already been delivered; The acquired incremental data is sent to the coordinating node.

2. The data synchronization method of claim 1, wherein, The steps to retrieve one incremental data entry from each source database include: Parse the log tables of each source database; Retrieve incremental data with global tags from the parsed log table.

3. The data synchronization method of claim 2, wherein, The global marker is a timestamp. The steps for determining the highest priority incremental data in the coordination node include: Compare the timestamps of the incremental data in the coordination node; The incremental data with the smallest timestamp is selected as the highest priority incremental data.

4. The data synchronization method of claim 2, wherein, The global marker is either a change number or a sequence number. The steps for determining the highest priority incremental data in the coordination node include: Convert the change number or sequence number of each incremental data in the coordination node into a timestamp. Compare the timestamps of the incremental data in the coordination node; The incremental data with the smallest timestamp is selected as the highest priority incremental data.

5. The data synchronization method of claim 1, wherein, After determining whether incremental data still exists in the source database where incremental data has been delivered, the process also includes: If no incremental data exists, a virtual data entry is returned to the coordinating node, and the priority of the virtual data entry is lower than the priority of the incremental data already existing in the coordinating node.

6. The data synchronization method according to claim 5, wherein, The timestamp of the virtual data is set to the timestamp when it is returned to the coordinating node.

7. The data synchronization method of claim 1, wherein, The steps of delivering the highest priority incremental data to the target database include: The highest priority incremental data is sent to the execution node, which is located between the coordinating node and the target database. The highest priority incremental data is processed and then delivered to the target database after processing.

8. A machine-readable storage medium having a machine-executable program stored thereon, the machine-executable program, when executed by a processor, implementing the data synchronization method according to any one of claims 1 to 7.

9. A computer device comprising a memory, a processor, and a machine-executable program stored in the memory and running on the processor, wherein the processor, when executing the machine-executable program, implements the data synchronization method according to any one of claims 1 to 7.

Images