Master-slave data synchronization optimization method and system

Abstract

The invention relates to the technical field of database optimization, in particular to a master-slave data synchronization optimization method and system.The method comprises the steps that a one-master multi-slave cluster of a database is created, and the cluster is provided with at least two slave nodes; creating a monitoring program, wherein the monitoring program can count the master-slave synchronization delay time of each pair; when the master node has a large data volume write-in operation or executes a large transaction, the slave nodes can be triggered to pull a large amount of data from the master node, at the moment, a synchronous delay threshold value is set, master-slave data synchronization is performed according to the threshold value, and the slave nodes with low synchronous delay are preferentially synchronized; according to the method, the influence of master-slave synchronization can be reduced to the greatest extent, and master-slave synchronization can be quickly completed by the slave nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database optimization, in particular to a master-slave data synchronization optimization method and system. BACKGROUND

[0002] At present, there are many high-concurrency and high-throughput scenarios. At the same time, due to the large number of users, the amount of user data also increases sharply. In order to ensure the read-write efficiency of the database, the database is usually deployed in a cluster manner, usually in a master-slave manner. The master node supports data writing, and the slave node supports reading and writing. Through the master-slave separation manner, the efficiency of the database and the system is improved. However, under the old manner, if the master-slave cluster database manner is used, after the master node writes a large amount of data, the multiple slave nodes also quickly pull information from the master node, which aggravates the IO of the master node. At this time, the efficiency of the slave node pulling data is reduced. At this time, most of the latest data of the multiple slave nodes is not synchronized successfully. Even if the multiple slave nodes provide services at the same time, these latest data cannot be read. SUMMARY

[0003] In order to overcome the problem that the IO read-write ability of the master node is reduced when the multiple slave nodes synchronize data, the efficiency of the slave node pulling data is reduced, and a large amount of latest data cannot be synchronized to the slave node in the prior art, the purpose of the present application is to provide a master-slave data synchronization optimization method and system, which can minimize the influence of master-slave synchronization and make the slave node complete the master-slave synchronization faster.

[0004] The present application adopts the following scheme to realize:

[0005] A master-slave data synchronization optimization method, the method steps are as follows:

[0006] Step 1: create a database master-slave cluster, the cluster has at least 2 slave nodes;

[0007] Step 2: create a monitoring program, the monitoring program can count the master-slave synchronization delay time of each pair;

[0008] Step 3: when the master node has a large amount of write operation or executes a large transaction, the slave node will pull a large amount of data from the master node. At this time, a synchronization delay threshold is set. If the master-slave synchronization delay of all slave nodes does not exceed the threshold, no processing is performed;

[0009] If the synchronization delay of the slave node exceeds the threshold, the slow slave node is actively disconnected, so that the slave nodes that do not exceed the threshold can quickly realize synchronization. After the slave nodes that do not exceed the threshold are synchronized, the synchronization of other slave nodes is started again;

[0010] If all the from nodes are all over the threshold value, at this time, the from nodes are synchronized in turn from low to high according to the size of the synchronization delay.

[0011] Further, the master-slave synchronization delay time is counted in seconds.

[0012] An optimization system for master-slave data synchronization, the system comprises a cluster creation module, a monitoring module, a master-slave data synchronization module;

[0013] The cluster creation module is used for creating a database one master and multiple froms cluster, and the cluster has at least two from nodes;

[0014] The monitoring module is used for creating a monitoring program, and the monitoring program can count the master-slave synchronization delay time of each pair;

[0015] The master-slave data synchronization module is used for triggering the from nodes to pull a large amount of data from the master node when the master node has a large amount of write operation or a large transaction is executed, at this time, a synchronization delay threshold value is set, if all the from nodes do not exceed the threshold value, no processing is performed;

[0016] If the synchronization delay of the from node exceeds the threshold value, the slow from node is actively disconnected, and the from nodes that do not exceed the threshold value can quickly realize synchronization, and when the from nodes that do not exceed the threshold value are synchronized, the synchronization of other from nodes is started again;

[0017] If all the from nodes are all over the threshold value, at this time, the from nodes are synchronized in turn from low to high according to the size of the synchronization delay.

[0018] Further, the master-slave synchronization delay time is counted in seconds.

[0019] The present application has the beneficial effects that:

[0020] The present application provides an optimization method and system for master-slave data synchronization, the delay amount of master-slave synchronization is monitored through a monitoring program, when the delay amount of a certain node is greatly different, the node is limitedly removed, and the master-slave synchronization is not performed, the node with better resources is preferentially ensured to complete synchronization and provide services to the outside. After the services are completed, the synchronization of other nodes is performed again. In this way, the influence of master-slave synchronization can be maximally reduced, and the from nodes can be quickly completed. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The flowchart of the method of the present application;

[0022] Figure 2 The structural block diagram of the system of the present application. DETAILED DESCRIPTION

[0023] The application will be further described in conjunction with the accompanying drawings.

[0024] Referring to Figure 1 An optimization method for master-slave data synchronization, the method comprising the following steps:

[0025] Step 1: creating a database cluster with one master and multiple slaves, the cluster having at least two slave nodes;

[0026] Step 2: creating a monitoring program capable of counting the synchronization delay time of each pair of master-slave;

[0027] Step 3: when the master node has a large amount of write operation or executes a large transaction, a large amount of data will be pulled from the master node by the slave nodes, at this time, a synchronization delay threshold is set, if the synchronization delay of all slave nodes does not exceed the threshold, no processing is performed;

[0028] If the synchronization delay of a slave node exceeds the threshold, the slow slave node is actively disconnected, ensuring that the slave nodes not exceeding the threshold can quickly realize synchronization, and after the synchronization of the slave nodes not exceeding the threshold is completed, the synchronization of other slave nodes is started;

[0029] If all slave nodes exceed the threshold, at this time, the slave nodes are synchronized in turn from low to high according to the size of the synchronization delay.

[0030] The application will be further described in conjunction with a specific embodiment:

[0031] An optimization method for master-slave data synchronization, the method comprising the following steps:

[0032] Step 1: creating a database cluster with one master and multiple slaves, the cluster having at least two slave nodes;

[0033] Step 2: creating a monitoring program capable of counting the synchronization delay time of each pair of master-slave;

[0034] Step 3: when the master node has a large amount of write operation or executes a large transaction, a large amount of data will be pulled from the master node by the slave nodes, at this time, a synchronization delay threshold is set, if the synchronization delay of all slave nodes does not exceed the threshold, no processing is performed;

[0035] If the synchronization delay of a slave node exceeds the threshold, the slow slave node is actively disconnected, ensuring that the slave nodes not exceeding the threshold can quickly realize synchronization, and after the synchronization of the slave nodes not exceeding the threshold is completed, the synchronization of other slave nodes is started;

[0036] For example: two slave nodes, one fast and one slow. When the synchronization delay of the slow slave node exceeds the threshold, the slow slave node can be actively disconnected at this time, ensuring that the fast slave node can quickly achieve synchronization. After the synchronization is completed, the synchronization of other slow slave nodes is started.

[0037] The active disconnection of the slow slave node is that the slave node no longer pulls data from the master node

[0038] If all slave nodes exceed the threshold, the slave nodes are synchronized in order from low to high according to the size of the synchronization delay at this time.

[0039] For example: two slow slave nodes, when the synchronization delay of the two slow slave nodes exceeds the threshold, the slower slave node can be disconnected, and the remaining steps are consistent with step 2.

[0040] There are two ways during subsequent synchronization. When the master-slave synchronization delay time of some nodes is within a short time, such as within 6-12 hours, the master-slave complete synchronization can be achieved slowly through subsequent synchronization, and the continue synchronization scheme is used.

[0041] Otherwise, when the master-slave synchronization time difference is large, the data is realized by copying the master node data file to the slave node and starting synchronization again.

[0042] An optimization system for master-slave data synchronization, the system comprises: a cluster creation module, a monitoring module, and a master-slave data synchronization module.

[0043] The cluster creation module is used to create a database master-slave cluster, and the cluster has at least two slave nodes.

[0044] The monitoring module is used to create a monitoring program, which can count the master-slave synchronization delay time of each pair.

[0045] The master-slave data synchronization module is used to trigger a large amount of data pulling from the master node to the slave node when the master node has a large amount of write operation or executes a large transaction. At this time, a synchronization delay threshold is set. If the synchronization delay of all slave nodes does not exceed the threshold, no processing is performed.

[0046] If the synchronization delay of some slave nodes exceeds the threshold, the slow slave node is actively disconnected, ensuring that the slave nodes that do not exceed the threshold can quickly achieve synchronization. After the synchronization of the slave nodes that do not exceed the threshold is completed, the synchronization of other slave nodes is started.

[0047] If all slave nodes exceed the threshold, the slave nodes are synchronized in order from low to high according to the size of the synchronization delay at this time.

[0048] In one embodiment of the application, the master-slave synchronization delay time is counted in seconds.

[0049] The above description is only the preferred embodiment of the present application, and any equivalent changes and modifications made according to the scope of the patent application of the present application shall be within the scope of the present application.

Claims

1. A master-slave data synchronization optimization method, characterized in that: The method steps are as follows: Step 1: Create a database cluster with one master and multiple slaves, with at least two slave nodes; Step 2: Create a monitoring program that can count the master-slave synchronization delay time of each pair; Step 3: When the master node writes a large amount of data or executes a large transaction, it triggers the slave nodes to pull a large amount of data from the master node. At this time, a synchronization delay threshold is set. If the master-slave synchronization delay of all slave nodes does not exceed the threshold, no processing is performed; If the synchronization delay of a slave node exceeds the threshold, the slow slave node is actively disconnected to ensure that the slave nodes within the threshold can quickly achieve synchronization. After the synchronization of the slave nodes within the threshold is completed, the synchronization of other slave nodes will be started; If all slave nodes exceed the threshold, the slave nodes will be synchronized in order from low to high according to the size of the synchronization delay.

2. The optimization method for master-slave data synchronization according to claim 1, characterized in that: The master-slave synchronization delay time is counted in seconds.

3. An optimization system for master-slave data synchronization, characterized in that: The system includes: a cluster creation module, a monitoring module, and a master-slave data synchronization module; The cluster creation module is used to create a database cluster with one master and multiple slaves, and the cluster has at least two slave nodes; The monitoring module is used to create a monitoring program, which can count the master-slave synchronization delay time of each pair; The master-slave data synchronization module is used to trigger the slave nodes to pull a large amount of data from the master node when there is a large amount of data write operation on the master node or after a large transaction is executed. At this time, a synchronization delay threshold is set. If the master-slave synchronization delay of all slave nodes does not exceed the threshold, no processing is performed; If the synchronization delay of a slave node exceeds the threshold, the slow slave node is actively disconnected to ensure that the slave nodes within the threshold can quickly achieve synchronization. After the synchronization of the slave nodes within the threshold is completed, the synchronization of other slave nodes will be started; If all slave nodes exceed the threshold, the slave nodes will be synchronized in order from low to high according to the size of the synchronization delay.

4. The master-slave data synchronization optimization system according to claim 3, characterized in that: The master-slave synchronization delay time is counted in seconds.

Images