A method for data synchronization between database clusters

Abstract

The application discloses a data synchronization method between database clusters, comprising the following steps: binding a floating address of a database cluster to a database master node in real time; starting a synchronization operation and maintenance process, a change capture process and a synchronization storage process on a node with the floating address in the database cluster, and closing the processes on a node without the floating address; parsing an archive log according to a last parsed archive log name and a position of the archive log, obtaining database change data and storing the change data in the database; monitoring changes of the change data in the database in real time, and sending the change data to a destination; starting a data packet receiving thread at the destination to complete data synchronization between a source and the destination; and completing data synchronization node and data synchronization strategy configuration through a data synchronization operation and maintenance monitoring module, and realizing data synchronization state monitoring, synchronization topology structure display and synchronization transaction watching.

Description

Technical Field

[0001] This invention relates to a data synchronization method, and more particularly to a data synchronization method between database clusters. Background Technology

[0002] With the continuous development and deepening of research on command information systems, their application areas are constantly expanding, and the scale of basic data is growing larger and larger. Highly reliable and near real-time data synchronization processing has become an urgent and inevitable requirement. At present, on the one hand, traditional change data capture methods based on SELECT (timestamp, flag, trigger, full table comparison, full extraction) have specific requirements for the database data structure, are intrusive to the database, or require scanning the entire table; on the other hand, traditional single-machine database data synchronization has single points of failure or lacks lightweight database cluster switchover tracking capabilities; at the same time, existing database synchronization services are not yet perfect in terms of unified encapsulation of heterogeneous data type mapping, and cannot meet the development requirements of high-level and diverse data synchronization processing. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a data synchronization method between database clusters, addressing the shortcomings of the prior art.

[0004] To address the aforementioned technical problems, this invention discloses a data synchronization method between database clusters, comprising the following steps:

[0005] Step 1: Bind the floating address of the database cluster to the database master node in real time. The floating address of the database cluster switches as the master and backup nodes of the database cluster switch.

[0006] Step 2: Start the synchronization operation and maintenance process, change capture process, and synchronization entry process on the nodes with floating addresses in the database cluster, and shut down the synchronization operation and maintenance process, change capture process, and synchronization entry process on the nodes without floating addresses.

[0007] Step 3: Based on the archived log name and location of the archived log from the previous parsing, parse the database archived log, obtain the database change data, and store the changed data in the database;

[0008] Step 4: Monitor changes in the database in real time, encapsulate the database changes into a change data message according to the preset message format, and send the change data message to the destination.

[0009] Step 5: The destination starts a data packet receiving thread. According to the preset packet format, it first parses the data packet, then constructs a data structure based on the changed data obtained after parsing and adds it to the cache list. Finally, it synchronizes the data in the cache list into the database to complete the data synchronization between the source and destination.

[0010] Step 6: Configure the data synchronization nodes and data synchronization strategies through the data synchronization operation and maintenance monitoring module, and realize data synchronization status monitoring, synchronization topology display and synchronization transaction viewing.

[0011] Furthermore, step 1 includes:

[0012] Step 1-1: Install the high availability service package on all database cluster nodes to provide high availability services;

[0013] Steps 1-2: Write a monitoring script program and run it on each node of the database cluster to determine whether the current node is the master node or the backup node of the database cluster.

[0014] Steps 1-3: Configure high availability services for each node in the database cluster, set the running mode to slave mode, and restart the high availability services for all nodes;

[0015] Steps 1-4: Based on the judgment results of the monitoring scripts on each node, the results are broadcast to other nodes in the database cluster in real time, and a master node is selected.

[0016] Furthermore, step 2 includes:

[0017] Step 2-1: Write the data synchronization daemon code and run it on each node of the database cluster. It is used to start the synchronization operation and maintenance process, change capture process and synchronization entry process on the current node after determining that the current node has a floating address.

[0018] Step 2-2: Write a scheduled task to check if the data synchronization daemon process exists at preset intervals; if it does not exist, run the data synchronization daemon process.

[0019] Steps 2-3: The synchronization daemon determines in real time whether the current node has a floating address. If the current node has a floating address, it captures the changes of the node, synchronizes them into the database, and starts the synchronization operation and maintenance process. Otherwise, it shuts down the changes capture, synchronization into the database, and synchronization operation and maintenance process of the node.

[0020] Furthermore, step 3 includes:

[0021] Step 3-1: Obtain the name of the archived log file and the SCN number of the log file it belongs to from the previous parsing;

[0022] Step 3-2: Based on the previous archived log name and SCN number, and the largest SCN number in the archived log at the current time, load the archived log file and query the change data from the previous time to the current time by the range of SCN numbers.

[0023] Step 3-3: Parse the changed data according to the preset format, perform data type conversion based on the preset unified data type, construct the data structure, and store the changed data in the database.

[0024] Furthermore, step 4, which involves encapsulating the database change data into a change data message, specifically includes:

[0025] First, perform heterogeneous mapping and unified encapsulation on the database changes, then build a buffer list to be sent and batch construct the message body.

[0026] Furthermore, step 5 includes:

[0027] Step 5-1: Call the data transmission initialization interface to complete the data synchronization transmission initialization settings and start the message sending thread and message receiving thread;

[0028] Step 5-2: Call the unified send and receive interface to send and receive synchronization status messages, including node status messages and node startup messages, to complete the status confirmation between the source and destination ends.

[0029] Step 5-3: Send the change data message from the source to the destination.

[0030] Furthermore, step 6 includes:

[0031] Step 6-1: The receiving thread at the destination end receives the message information sent from the source end in real time and parses it according to the agreed method. The parsed data is used to construct a data structure and added to the database cache list.

[0032] Step 6-2: The receiving end's review thread starts working, updating the approved message data to the corresponding database table, completing the incremental data synchronization between databases.

[0033] Furthermore, step 7 includes:

[0034] Step 7-1: Add a data synchronization node to the data synchronization operation and maintenance monitoring module to complete the creation of data synchronization nodes at the source and destination ends;

[0035] Step 7-2: Add data synchronization strategies to the data synchronization operation and maintenance monitoring module, including homogeneous table strategy and heterogeneous table strategy configuration;

[0036] Step 7-3: The data synchronization operation and maintenance monitoring module is used to monitor the status of synchronization nodes, display the synchronization topology, and view synchronization transactions.

[0037] Furthermore, configuring the high availability service of each node in the database cluster as described in steps 1-3 includes at least configuring the real address, floating address, weight, and priority of the database cluster.

[0038] Furthermore, the specific method for selecting the master node as described in steps 1-3 is as follows:

[0039] According to the preset election mechanism, the process of selecting the master node is completed through a high-availability service.

[0040] Beneficial effects:

[0041] (1) The present invention adopts a lightweight high availability framework, dynamically monitors database cluster switching, and provides users with seamless data synchronization capabilities between database clusters, effectively improving the high availability of data synchronization processing.

[0042] (2) The present invention uses CDC log parsing technology to parse complete archived logs in real time, providing near real-time incremental change data capture capability, which effectively improves the accuracy of data synchronization processing.

[0043] (3) The present invention adopts a unified encapsulation design for heterogeneous data type mapping, which satisfies the data synchronization between multiple heterogeneous databases and effectively improves the diversity of data synchronization processing.

[0044] (4) The present invention provides a three-layer architecture based on capture-transmission-database. The three layers are independent of each other but interdependent, which effectively improves the adaptability of data synchronization processing.

[0045] (5) The various functional units of the synchronous operation and maintenance monitoring of this invention call each other through API interface, which facilitates subsequent expansion and development and improves the execution efficiency of the system. Attached Figure Description

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0047] Figure 1 This is a schematic diagram of the system architecture of the data synchronization method disclosed in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the system composition of the data synchronization method disclosed in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the lightweight, highly available module composition of the system of the present invention.

[0050] Figure 4 This is a schematic diagram of the composition of the synchronization process protection module of the system of the present invention.

[0051] Figure 5 This is a schematic diagram of the change data capture module of the system of the present invention.

[0052] Figure 6 This is a schematic diagram of the synchronous data transmission module of the system of the present invention.

[0053] Figure 7 This is a schematic diagram of the synchronous data entry module of the system of the present invention.

[0054] Figure 8 This is a schematic diagram of the synchronous operation and maintenance monitoring module of the system of the present invention. Detailed Implementation

[0055] To overcome the limitations of traditional processing methods, this paper considers a lightweight, high-availability framework and CDC log parsing technology on a high-concurrency, high-volume database cluster. By fully utilizing the near real-time and incremental change data capture advantages of CDC log parsing, this method employs lightweight dynamic monitoring database cluster switching technology to achieve database cluster change data capture, transmission, storage, and monitoring. This significantly improves the timeliness and accuracy of basic data synchronization in the command information system, laying a solid foundation for future data synchronization analysis and processing between database clusters.

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0057] like Figure 1 The diagram shown is a system architecture diagram of a data synchronization method between database clusters disclosed in an embodiment of the present invention, illustrating the data flow and hierarchical relationship of the method, as follows: Figure 2 The diagram shown is a system composition diagram of a data synchronization method between database clusters disclosed in an embodiment of the present invention, including the following modules: lightweight high availability module 1, synchronization process guardian module 2, change data capture module 3, synchronization data transmission module 4, synchronization data entry module 5, and synchronization operation and maintenance monitoring module 6.

[0058] Among them, such as Figure 3 As shown, the lightweight high availability module 1 includes a cluster configuration unit 7, a monitoring program unit 8, and a floating address unit 9. The lightweight high availability module first configures the real address, floating address, and monitoring program scheduling parameters of the database cluster in the configuration files of each node. Secondly, the monitoring program on each node reports the monitoring results in real time. Finally, through an election mechanism, the floating address switches with the switching of the database cluster master node, ensuring that the floating address is always bound to the database cluster master node.

[0059] like Figure 4As shown, the synchronization process guardian module 2 includes a floating address monitoring unit 10, a synchronization guardian unit 11, and an operation and maintenance guardian unit 12. The synchronization process guardian module determines in real time whether the current node has a floating address. If the current node has a floating address, it captures the changes of the node, synchronizes them into the database, and starts the synchronization operation and maintenance process. If the current node does not have a floating address, it shuts down the changes capture, synchronization into the database, and synchronization operation and maintenance process, ensuring that only nodes with floating addresses will start the relevant processes and provide services to the outside world.

[0060] like Figure 5 As shown, the change data capture module 3 includes a log file loading unit 13, a change data parsing unit 14, and a change data caching unit 15. The change data capture module uses CDC log parsing technology to extract the most original change data from the log file, performs corresponding data type conversions using a unified data type, and then stores the change data in the database for use by the synchronization data transmission module.

[0061] like Figure 6 As shown, the synchronous data transmission module 4 includes a unified transmission encapsulation unit 16, a message sending and receiving unit 17, and a batch data transmission unit 18. The synchronous data transmission module first encapsulates the underlying transmission service, providing unified interfaces for transmission initialization, data sending, and data receiving. Secondly, it calls the unified sending and receiving interface to send and receive synchronous status messages, including node status messages and node startup messages. Finally, it calls the unified sending and receiving interface to construct batch change data into data messages and send them from the source end to the destination end.

[0062] like Figure 7 As shown, the synchronous data entry module 5 includes a synchronous data caching unit 19, a synchronous data review unit 20, and a synchronous data entry unit 21. The synchronous data entry module first parses the message information received by the synchronous data transmission module according to a pre-defined method. The parsed data is used to construct a data structure and added to the cache list. Secondly, after the modified data passes review, the cached data is stored in the database, completing the data synchronization between databases.

[0063] like Figure 8 As shown, the synchronization operation and maintenance monitoring module 6 includes a synchronization node management unit 22, a synchronization configuration management unit 23, and a synchronization status monitoring unit 24. The synchronization operation and maintenance monitoring module first manages the synchronization nodes, then manages the homogeneous table strategy or heterogeneous table strategy between nodes, and finally monitors the synchronization node status, displays the synchronization topology, and allows for viewing synchronization transactions.

[0064] The workflow of this invention is as follows:

[0065] When using it, first complete the configuration and binding of the floating address of the database cluster, such as... Figure 1As shown, the floating address switches with the database cluster master node, ensuring that the floating address is always bound to the master node. Next, the synchronization table strategy or heterogeneous table synchronization strategy configuration between the source and destination nodes is completed, and full data initialization between synchronization nodes is performed as an initialization task. The source node change capture module periodically scans the database redo log to capture data changes, encapsulates the data changes into transactions, and stores them locally. The synchronization transmission module is responsible for reading data changes, scanning the transaction table in near real-time, and storing any new database change transactions in the synchronization transaction table, either actively or waiting for a request from the peer node. The source synchronization node's synchronization transmission module sends the database change information to the destination synchronization transmission module in near real-time. The destination synchronization entry module reads the transaction, locates the corresponding record, makes modifications, and enters the changed data into the database, while simultaneously sending the data synchronization completion transaction number to the source node. After receiving the synchronization completion transaction number from the destination node, the source node clears the synchronization transaction table to ensure that the transaction table does not become too large. If the source node fails to synchronize data with the destination node due to network issues or other reasons, the synchronization change transaction table information on the source node will persist until the source node detects that communication is normal and resynchronizes with the destination node. Any data change on either the source or destination node will be sent to the other end according to the above process and synchronized in the database on the other end, ensuring data consistency between any nodes.

[0066] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the content of the data synchronization method between database clusters provided by this invention, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0067] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0068] This invention provides a concept and method for data synchronization between database clusters. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for data synchronization between database clusters, characterized in that, Includes the following steps: Step 1: Bind the floating address of the database cluster to the database master node in real time. The floating address of the database cluster switches as the master and backup nodes of the database cluster switch. Step 2: Start the synchronization operation and maintenance process, change capture process, and synchronization entry process on the nodes with floating addresses in the database cluster, and shut down the synchronization operation and maintenance process, change capture process, and synchronization entry process on the nodes without floating addresses. Step 3: Based on the archived log name and location of the archived log from the previous parsing, parse the database archived log, obtain the database change data, and store the changed data in the database; Step 4: Monitor changes in the database in real time, encapsulate the database changes into a change data message according to the preset message format, and send the change data message to the destination. Step 5: The destination starts a data packet receiving thread. According to the preset packet format, it first parses the data packet, then constructs a data structure based on the changed data obtained after parsing and adds it to the cache list. Finally, it synchronizes the data in the cache list into the database to complete the data synchronization between the source and destination. Step 6: Configure the data synchronization nodes and data synchronization strategies through the data synchronization operation and maintenance monitoring module, and realize data synchronization status monitoring, synchronization topology display and synchronization transaction viewing.

2. The data synchronization method between database clusters according to claim 1, characterized in that, Step 1 includes: Step 1-1: Install the high availability service package on all database cluster nodes to provide high availability services; Steps 1-2: Write a monitoring script program and run it on each node of the database cluster to determine whether the current node is the master node or the backup node of the database cluster. Steps 1-3: Configure high availability services for each node in the database cluster, set the running mode to slave mode, and restart the high availability services for all nodes; Steps 1-4: Based on the judgment results of the monitoring scripts on each node, the results are broadcast to other nodes in the database cluster in real time, and a master node is selected.

3. The data synchronization method between database clusters according to claim 2, characterized in that, Step 2 includes: Step 2-1: Write the data synchronization daemon code and run it on each node of the database cluster. It is used to start the synchronization operation and maintenance process, change capture process and synchronization entry process on the current node after determining that the current node has a floating address. Step 2-2: Write a scheduled task to check if the data synchronization daemon process exists at preset intervals; if it does not exist, run the data synchronization daemon process. Steps 2-3: The synchronization daemon determines in real time whether the current node has a floating address. If the current node has a floating address, it captures the changes of the node, synchronizes them into the database, and starts the synchronization operation and maintenance process. Otherwise, it shuts down the changes capture, synchronization into the database, and synchronization operation and maintenance process of the node.

4. The data synchronization method between database clusters according to claim 3, characterized in that, Step 3 includes: Step 3-1: Obtain the name of the archived log file and the SCN number of the log file it belongs to from the previous parsing; Step 3-2: Based on the previous archived log name and SCN number, and the largest SCN number in the archived log at the current time, load the archived log file and query the change data from the previous time to the current time by the range of SCN numbers. Step 3-3: Parse the changed data according to the preset format, perform data type conversion based on the preset unified data type, construct the data structure, and store the changed data in the database.

5. The data synchronization method between database clusters according to claim 4, characterized in that, Step 4, which involves encapsulating the database change data into a change data message, specifically includes: First, perform heterogeneous mapping and unified encapsulation on the database changes, then build a buffer list to be sent and batch construct the message body.

6. The data synchronization method between database clusters according to claim 5, characterized in that, Step 5 includes: Step 5-1: Call the data transmission initialization interface to complete the data synchronization transmission initialization settings and start the message sending thread and message receiving thread; Step 5-2: Call the unified send and receive interface to send and receive synchronization status messages, including node status messages and node startup messages, to complete the status confirmation between the source and destination ends. Step 5-3: Send the change data message from the source to the destination.

7. The data synchronization method between database clusters according to claim 6, characterized in that, Step 6 includes: Step 6-1: The receiving thread at the destination end receives the message information sent from the source end in real time and parses it according to the agreed method. The parsed data is used to construct a data structure and added to the database cache list. Step 6-2: The receiving end's review thread starts working, updating the approved message data to the corresponding database table, completing the incremental data synchronization between databases.

8. The data synchronization method between database clusters according to claim 7, characterized in that, Step 7 includes: Step 7-1: Add a data synchronization node to the data synchronization operation and maintenance monitoring module to complete the creation of data synchronization nodes at the source and destination ends; Step 7-2: Add data synchronization strategies to the data synchronization operation and maintenance monitoring module, including homogeneous table strategy and heterogeneous table strategy configuration; Step 7-3: The data synchronization operation and maintenance monitoring module is used to monitor the status of synchronization nodes, display the synchronization topology, and view synchronization transactions.

9. A data synchronization method between database clusters according to claim 8, characterized in that, The high availability service configuration for each node of the database cluster described in steps 1-3 includes at least configuring the real address, floating address, weight, and priority of the database cluster.

10. A data synchronization method between database clusters according to claim 9, characterized in that, The specific method for selecting the master node as described in steps 1-3 is as follows: According to the preset election mechanism, the process of selecting the master node is completed through a high-availability service.

Images