Data synchronization method and device, storage medium and processor

Abstract

The present application discloses a data synchronization method and device, storage medium and processor. The method includes: obtaining multiple target data and multiple first transfer nodes through a second transfer node; determining the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; allocating each target data to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server. Through this application, the problem of poor data synchronization effect caused by the method of static binding of the source server in the cluster environment and the transfer node for processing data in the related art is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method and device, a storage medium, and a processor. Background Art

[0002] Cluster technology is a relatively new technology. Through cluster technology, relatively high benefits in performance, reliability, and flexibility can be obtained at a lower cost. Its task scheduling is the core technology in cluster systems.

[0003] Furthermore, some servers deployed in clusters often need to synchronize data to a target network. Because the interfaces provided by the data source server and the target server are incompatible, or the required data formats are inconsistent, a transit node is required to process the data before synchronization. This transit node can be either a large server or data processing software installed on a PC (Personal Computer). The following are three application scenarios:

[0004] (1) In the audit and control of network security, security vendors often deploy proxy software in the internal network to synchronize users from the authentication server deployed in the cluster. The proxy software forwards user data to the network security device to help the network security device complete the real-name audit and control of traffic.

[0005] (2) In a complex audit system, the logs of various devices are usually different. In order to aggregate the logs to the log platform for unified management, a transfer node can be set up to uniformly process and format the logs.

[0006] (3) In a small network, the cluster-deployed server 1 stores resources accessible to users, and server 2 is used to reverse proxy user requests and provide resources to users. A transfer node is set between server 1 and server 2 to perform data decoding, classification, formatting and other processing operations.

[0007] However, these relay nodes may have limited data processing capabilities and stability. To ensure synchronization efficiency and stability, we typically implement load balancing and redundant deployment, using multiple relay nodes to connect to multiple source servers in a cluster environment, process data, and forward it to the destination server.

[0008] Related technologies use a one-to-many data synchronization approach, using a single relay node to connect to multiple source servers in a cluster environment and synchronize data from the source servers to the target network. However, this approach suffers from insufficient processing power or instability on a single relay node, potentially leading to congestion or even complete paralysis of the data synchronization node.

[0009] In addition, related technologies also use a simple N-to-N method to synchronize data, that is, there are multiple transit nodes, each of which is bound to a source server and synchronizes the source server's data to the target network. However, the disadvantage of this method is that due to the binding relationship, redundant deployment is not convenient. In order to prevent a transit node from being paralyzed, a backup redundant node must be set up for it. When the number of paralyzed nodes is greater than the number of redundant nodes, synchronization will be interrupted or partially interrupted. In addition, due to the amount of data that each source server needs to synchronize, the actual carrying capacity of each transit node is different. The static binding relationship will cause the synchronization resources to be unevenly distributed, making some transit nodes too congested and some transit nodes too idle.

[0010] Currently, no effective solution has been proposed to the problem that the data of the source server in the cluster environment is synchronized to the destination server by statically binding the source server in the cluster environment with the transit node for processing data, resulting in poor data synchronization effect. Summary of the Invention

[0011] The main purpose of this application is to provide a data synchronization method and device, storage medium and processor to solve the problem that the related technology adopts the method of static binding of the source server in the cluster environment and the transit node for processing data to synchronize the data of the source server in the cluster environment to the destination server, resulting in poor data synchronization effect.

[0012] To achieve the above-mentioned object, according to one aspect of the present application, a data synchronization method is provided. The method is applied in a data synchronization system, the data synchronization system comprising at least: a plurality of first transfer nodes and a second transfer node, the first transfer node being used to process data, the second transfer node being used to schedule the plurality of first transfer nodes, if there is a faulty node among the plurality of first transfer nodes, a transfer node other than the faulty node among the first transfer nodes is used to replace the faulty node, if the second transfer node fails, any one of the plurality of first transfer nodes is used to replace the second transfer node, comprising: obtaining a plurality of target data and the plurality of first transfer nodes through the second transfer node, wherein each target data is data to be synchronized to a destination server from a plurality of source servers in a cluster environment; determining the number of the plurality of target data and the efficiency of each first transfer node in processing the data through the second transfer node; allocating each target data to each first transfer node through the second transfer node based on the number of the plurality of target data and the efficiency of each first transfer node in processing the data, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronizing the processed plurality of target data to the destination server.

[0013] Furthermore, allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data includes: calculating the amount of data allocated to each first transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data; and allocating each target data to each first transfer node based on the amount of data allocated to each first transfer node.

[0014] Furthermore, before determining the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, the method also includes: determining multiple target transfer nodes, wherein the multiple target transfer nodes include at least the multiple first transfer nodes; determining the second transfer node from the multiple target transfer nodes to determine the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes other than the multiple first transfer nodes.

[0015] Furthermore, after allocating each target data to each first transit node through the second transit node based on the number of the multiple target data and the efficiency of each first transit node in processing data, the method also includes: judging whether there is a faulty node among the multiple first transit nodes; if there is a faulty node among the multiple first transit nodes, determining multiple third transit nodes other than the faulty node from the multiple first transit nodes, wherein the third transit nodes are used to process the multiple target data instead of the first transit node; obtaining the efficiency of each third transit node in processing data; and allocating each target data to each third transit node based on the number of the multiple target data and the efficiency of each third transit node in processing data.

[0016] Furthermore, after determining the second transfer node from the multiple target transfer nodes, the method also includes: determining whether the second transfer node fails; if the second transfer node fails, determining a fourth transfer node from the multiple first transfer nodes, wherein the fourth transfer node is used to replace the second transfer node to schedule the transfer nodes among the multiple first transfer nodes except the fourth transfer node.

[0017] Furthermore, allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data includes: judging whether the data volume of the multiple target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, so that the first transfer node obtains the multiple target data from the multiple source servers and processes the multiple target data; if the data volume of the multiple target data is not greater than the preset data volume, obtaining the multiple target data from the multiple source servers through the second transfer node; and allocating each target data obtained from the multiple source servers to each first transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data.

[0018] Furthermore, determining the second transfer node from the multiple target transfer nodes includes: obtaining the numbering information corresponding to each target transfer node; determining the transfer node with the smallest number among the multiple target transfer nodes based on the numbering information corresponding to each target transfer node; and using the transfer node with the smallest number among the multiple target transfer nodes as the second transfer node.

[0019] Furthermore, the method also includes: obtaining the amount of data to be synchronized to the destination server from each source server per unit time; calculating the average queue length of each first transfer node based on the amount of data to be synchronized to the destination server from each source server per unit time, wherein the average queue length is used to represent the accumulation of the multiple target data in the first time period; determining the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure of each first transfer node in the second time period, and the length of the second time period is less than the length of the first time period; sending a reminder message to the target object based on the average queue length and real-time queue length of each first transfer node, wherein the reminder message is used to remind the target object to adjust the strategy of synchronizing the multiple target data to the destination server.

[0020] Furthermore, after obtaining multiple target data and multiple first transfer nodes, the method also includes: determining whether the fifth transfer node among the multiple first transfer nodes is in an idle state; if the fifth transfer node among the multiple first transfer nodes is in an idle state, allocating the first data among the multiple target data to the fifth transfer node, so that the fifth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server; if the fifth transfer node among the multiple first transfer nodes is not in an idle state, determining whether the sixth transfer node among the multiple first transfer nodes is in an idle state; if the sixth transfer node among the multiple first transfer nodes is in an idle state, allocating the first data among the multiple target data to the sixth transfer node, so that the sixth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server.

[0021] In order to achieve the above object, according to another aspect of the present application, a data synchronization device is provided. The device is applied in a data synchronization system, which at least includes: a plurality of first transfer nodes and a second transfer node, wherein the first transfer node is used to process data, and the second transfer node is used to schedule the plurality of first transfer nodes. If a faulty node exists among the plurality of first transfer nodes, a transfer node other than the faulty node among the first transfer nodes is used to replace the faulty node. If the second transfer node fails, any one of the plurality of first transfer nodes is used to replace the second transfer node. The device includes: a first acquisition module, which is used to acquire a plurality of target data and the plurality of first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from a plurality of source servers to a destination server in a cluster environment; a first determination module, which is used to determine the number of the plurality of target data and the data processing efficiency of each first transfer node through the second transfer node; and a first processing module, which is used to allocate each target data to each first transfer node through the second transfer node according to the number of the plurality of target data and the data processing efficiency of each first transfer node, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronize the processed plurality of target data to the destination server.

[0022] Furthermore, the first processing module includes: a first calculation unit, used to calculate the amount of data allocated to each first transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data; a first allocation unit, used to allocate each target data to each first transfer node based on the amount of data allocated to each first transfer node.

[0023] Furthermore, the device also includes: a second determination module, used to determine multiple target transfer nodes before determining the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the multiple target transfer nodes include at least the multiple first transfer nodes; a third determination module, used to determine the second transfer node from the multiple target transfer nodes, so as to determine the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes except the multiple first transfer nodes.

[0024] Furthermore, the device also includes: a first judgment module, used to judge whether there is a faulty node among the multiple first transfer nodes after each target data is allocated to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data; a fourth determination module, used to determine a plurality of third transfer nodes other than the faulty node from the multiple first transfer nodes if there is a faulty node among the multiple first transfer nodes, wherein the third transfer node is used to process the multiple target data instead of the first transfer node; a second acquisition module, used to acquire the efficiency of each third transfer node in processing data; a first allocation module, used to allocate each target data to each third transfer node based on the number of the multiple target data and the efficiency of each third transfer node in processing data.

[0025] Furthermore, the device also includes: a second judgment module, used to judge whether the second transfer node fails after determining the second transfer node from the multiple target transfer nodes; a fifth determination module, used to determine a fourth transfer node from the multiple first transfer nodes if the second transfer node fails, wherein the fourth transfer node is used to replace the second transfer node to schedule the transfer nodes among the multiple first transfer nodes except the fourth transfer node.

[0026] Furthermore, the first processing module includes: a first judgment unit, used to judge whether the data volume of the multiple target data is greater than the preset data volume; a second allocation unit, used to allocate each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data if the data volume of the multiple target data is greater than the preset data volume, so that the first transfer node obtains the multiple target data from the multiple source servers and processes the multiple target data; a first acquisition unit, used to obtain the multiple target data from the multiple source servers through the second transfer node if the data volume of the multiple target data is not greater than the preset data volume; a third allocation unit, used to allocate each target data obtained from the multiple source servers to each first transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data.

[0027] Furthermore, the third determination module includes: a second acquisition unit, used to obtain the numbering information corresponding to each target transit node; a first determination unit, used to determine the transit node with the smallest number among the multiple target transit nodes based on the numbering information corresponding to each target transit node; and a second determination unit, used to use the transit node with the smallest number among the multiple target transit nodes as the second transit node.

[0028] Furthermore, the device also includes: a third acquisition module, used to obtain the amount of data to be synchronized to the destination server from each source server per unit time; a first calculation module, used to calculate the average queue length of each first transfer node based on the amount of data to be synchronized to the destination server from each source server per unit time, wherein the average queue length is used to represent the accumulation of the multiple target data in the first time period; a sixth determination module, used to determine the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure of each first transfer node in the second time period, and the length of the second time period is less than the length of the first time period; a first sending module, used to send a reminder message to the target object based on the average queue length and real-time queue length of each first transfer node, wherein the reminder message is used to remind the target object to adjust the strategy of synchronizing the multiple target data to the destination server.

[0029] Furthermore, the device also includes: a third judgment module, which is used to determine whether the fifth transfer node among the multiple first transfer nodes is in an idle state after obtaining multiple target data and multiple first transfer nodes; a second allocation module, which is used to allocate the first data among the multiple target data to the fifth transfer node if the fifth transfer node among the multiple first transfer nodes is in an idle state, so that the fifth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server; a fourth judgment module, which is used to determine whether the sixth transfer node among the multiple first transfer nodes is in an idle state if the fifth transfer node among the multiple first transfer nodes is not in an idle state; and a third allocation module, which is used to allocate the first data among the multiple target data to the sixth transfer node if the sixth transfer node among the multiple first transfer nodes is in an idle state, so that the sixth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server.

[0030] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein the program executes any one of the above-mentioned data synchronization methods.

[0031] In order to achieve the above-mentioned object, according to another aspect of the present application, a processor is provided, wherein the processor is used to run a program, wherein the program executes any one of the above-mentioned data synchronization methods when running.

[0032] The present application adopts the following steps: obtaining multiple target data and multiple first transfer nodes through a second transfer node, wherein each target data is data to be synchronized to a destination server from multiple source servers in a cluster environment; determining the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; and assigning each target data to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, thereby solving the problem of poor data synchronization effect caused by the method of static binding of source servers in a cluster environment and transfer nodes for processing data in the related art. By assigning each target data to each first transfer node based on the determined number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, the efficiency and stability of data synchronization are improved, thereby improving the effect of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0034] Figure 1 is a flowchart of a data synchronization method provided in accordance with an embodiment of the present application;

[0035] Figure 2 is a schematic diagram of a deployment data synchronization device provided according to an embodiment of the present application;

[0036] Figure 3 This is a schematic diagram of a scheduling node forwarding task data when the data to be processed occupies a large amount of bandwidth in an embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of a scheduling node forwarding task data when the data to be processed does not need to occupy a large amount of bandwidth in an embodiment of the present application;

[0038] Figure 5 The data synchronization method according to the embodiment of the present application is provided. Figure 1 ;

[0039] Figure 6 is a flowchart of an optional data synchronization method provided according to an embodiment of the present application;

[0040] Figure 7 2 is a schematic diagram of a data synchronization device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] Example 1

[0045] The present invention will be described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a data synchronization method provided in accordance with an embodiment of the present application. Figure 1 As shown, the method is applied in a data synchronization system, which includes at least: multiple first transfer nodes and second transfer nodes, the first transfer nodes are used to process data, and the second transfer nodes are used to schedule the multiple first transfer nodes. If there is a faulty node among the multiple first transfer nodes, a transfer node other than the faulty node among the first transfer nodes is used to replace the faulty node. If the second transfer node fails, any one of the multiple first transfer nodes is used to replace the second transfer node, and the method includes the following steps:

[0046] Step S101: Acquire multiple target data and multiple first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from multiple source servers to a destination server in a cluster environment.

[0047] For example, Figure 2 is a schematic diagram of a data synchronization device deployed according to an embodiment of the present application. Figure 2 As shown, the number of destination servers in the data synchronization device can be multiple. Since the embodiment of the present application does not focus on the binding relationship between the destination server and each transfer node, the destination server can be regarded as one.

[0048] For example, assume there are m source servers and n transit nodes. The transit nodes synchronize data from the source servers, process it, and send it to the destination server.

[0049] In addition, among all the transit nodes, one is considered a scheduling node (the second transit node mentioned above). The scheduling node is responsible for scheduling all transit nodes, while the other nodes are responsible for data processing, also known as data processing nodes (the first transit node mentioned above). For any transit node, its internal structure consists of a network module and a multi-process data processing module. The former is responsible for sending and receiving data, and the latter is responsible for data processing. There is no difference between a scheduling node and a data processing node. They are only selected during actual operation to schedule other nodes. If necessary, any transit node can serve as a scheduling node. In other words, if one or more data processing nodes fail, the scheduling node can record the failed data processing nodes and select a healthy data processing node to take over the work of the failed data processing node. If the current scheduling node fails, a new scheduling node can be elected from the data processing nodes in order of number. For example, if the scheduling node numbered 1 fails, the node numbered 2 becomes the new scheduling node. If the scheduling node numbered 2 fails, the node numbered 3 becomes the new scheduling node.

[0050] For example, the scheduling node determines the data that needs to be synchronized from multiple source servers in the cluster environment to the destination server (the multiple target data mentioned above), and determines which data processing nodes exist in the data synchronization device.

[0051] Step S102: determining the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data through the second transfer node.

[0052] For example, the scheduling node determines the amount of data that needs to be synchronized from multiple source servers in the cluster environment to the destination server, and calculates the amount of data processed by each data processing node per unit time, that is, the average processing rate (the efficiency of each first transit node in processing data mentioned above).

[0053] Step S103, through the second transfer node, each target data is allocated to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server.

[0054] For example, the scheduling node allocates the data to be synchronized to each data processing node based on the amount of data to be synchronized from multiple source servers in the cluster environment to the destination server and the amount of data processed per unit time by each data processing node (average processing rate). After receiving the assigned data, each data processing node processes the data to obtain the processed data, which it then synchronizes to the destination server.

[0055] Through the above steps S101 to S103, the second transfer node allocates each target data to each first transfer node based on the determined number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

[0056] In order to quickly and accurately determine the second transfer node, in the data synchronization method provided in the embodiment of the present application, the second transfer node can also be determined by the following steps: determining multiple target transfer nodes, wherein the multiple target transfer nodes include at least multiple first transfer nodes; determining the second transfer node from the multiple target transfer nodes, so as to determine the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes except the multiple first transfer nodes.

[0057] For example, Figure 2 As shown, there may be multiple transfer nodes (corresponding to Figure 2 Transfer node 1, transfer node 2, transfer node 3, transfer node 4, etc.), and Figure 2The transfer nodes 1, 2, 3, and 4 in the above-mentioned transfer nodes can be the target transfer nodes mentioned above. Then, any one of the transfer nodes such as transfer node 1, transfer node 2, transfer node 3, and transfer node 4 can be selected as the scheduling node (the second transfer node mentioned above), and the scheduling node determines the amount of data that needs to be synchronized from multiple source servers in the cluster environment to the destination server, and calculates the amount of data processed by each data processing node per unit time, that is, the average processing rate (the efficiency of each first transfer node in processing data mentioned above). In addition, if the transfer node 1 among the transfer nodes such as transfer node 1, transfer node 2, transfer node 3, and transfer node 4 is used as the scheduling node, the remaining transfer nodes such as transfer node 2, transfer node 3, and transfer node 4 are used as data processing nodes (the multiple first transfer nodes mentioned above).

[0058] To summarize, a multi-node synchronization device is designed, and one node is selected as the scheduling node. The scheduling node is used to uniformly connect to each server in the cluster, and the processing node completes the processing work based on load balancing. It is easy to deploy and can avoid insufficient load on a single node.

[0059] In order to quickly and accurately determine the second transit node, in the data synchronization method provided in the embodiment of the present application, the second transit node can also be determined by the following steps: obtaining the numbering information corresponding to each target transit node; based on the numbering information corresponding to each target transit node, determining the transit node with the smallest number among multiple target transit nodes; and using the transit node with the smallest number among multiple target transit nodes as the second transit node.

[0060] For example, the transit nodes can be automatically numbered according to the time they join the system, and the transit node numbered 1 can be automatically elected as the scheduling node. The scheduling node is responsible for node scheduling and task allocation, but not for data processing.

[0061] Through the above solution, the scheduling node can be determined quickly and accurately from multiple transfer nodes.

[0062] In order to quickly and accurately assign each target data to each first transfer node, in the data synchronization method provided in the embodiment of the present application, each target data can also be assigned to each first transfer node through the following steps: determine whether the data volume of multiple target data is greater than the preset data volume; if the data volume of multiple target data is greater than the preset data volume, then through the second transfer node, each target data is assigned to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that the first transfer node obtains multiple target data from multiple source servers and processes the multiple target data; if the data volume of multiple target data is not greater than the preset data volume, then through the second transfer node, multiple target data are obtained from multiple source servers; based on the number of multiple target data and the efficiency of each first transfer node in processing data, each target data obtained from multiple source servers is assigned to each first transfer node.

[0063] For example, when the data to be processed occupies a large amount of bandwidth, and the transmission capacity of the scheduling node is limited, when the scheduling node assigns tasks, the process is as follows: the scheduling node obtains task-related information from the source server, such as the data number, etc., without actually synchronizing the data. When the data synchronization work needs to be assigned to other transit nodes, a message is sent to inform the other transit nodes of the source server, data number, quantity, and other information corresponding to the task, and the transit node obtains the data from the source server on its own. And when the data to be processed occupies a large amount of bandwidth, the scheduling node forwards the task data, as shown in the following diagram: Figure 3 shown.

[0064] For example, when the data to be processed does not require a large amount of bandwidth and the scheduling node has sufficient transmission capacity, the scheduling node performs task allocation in the following manner: the scheduling node obtains task-related information and the data itself from the source server. When data synchronization work needs to be allocated to other transfer nodes, a message is sent to forward the data to the other transfer nodes. At this time, there is no need for the other transfer nodes to establish a connection with the source server. And when the data to be processed does not require a large amount of bandwidth, the scheduling node performs task data forwarding, as shown in the following diagram: Figure 4 shown.

[0065] In summary, two deployment modes can be set according to the type of data being synchronized. When the data occupies a large amount of bandwidth, the scheduling node performs task assignment. When the data does not occupy a large amount of bandwidth, the scheduling node performs actual task data distribution.

[0066] In order to quickly and accurately obtain the processed first data, in the data synchronization method provided in the embodiment of the present application, the processed first data can also be obtained by the following steps: determine whether the fifth transfer node among the multiple first transfer nodes is in an idle state; if the fifth transfer node among the multiple first transfer nodes is in an idle state, the first data among the multiple target data is allocated to the fifth transfer node, so that the fifth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server; if the fifth transfer node among the multiple first transfer nodes is not in an idle state, determine whether the sixth transfer node among the multiple first transfer nodes is in an idle state; if the sixth transfer node among the multiple first transfer nodes is in an idle state, the first data among the multiple target data is allocated to the sixth transfer node, so that the sixth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server.

[0067] For example, during the initial system operation, a relatively simple round-robin load balancing approach is employed. The process is as follows: the dispatching node retrieves tasks from various source servers, traversing through other relay nodes each time and allocating a fixed amount of tasks to each relay node. Before allocating tasks, the dispatching node checks whether the target relay node is idle. If so, the task is assigned to that relay node. If not, the dispatching node is queried for the next available relay node.

[0068] Through the above solution, the scheduling node can initially quickly distribute the data to be synchronized to each source server for processing according to the polling load balancing method.

[0069] Figure 5 The data synchronization method according to the embodiment of the present application is provided. Figure 1 ,like Figure 5 As shown, in the data synchronization method provided in the embodiment of the present application, allocating each target data to each first transfer node by the second transfer node according to the number of multiple target data and the efficiency of each first transfer node in processing data includes:

[0070] Step S501, calculating the amount of data allocated to each first transfer node based on the amount of multiple target data and the efficiency of each first transfer node in processing data;

[0071] Step S502: Allocate each target data to each first transfer node based on the amount of data allocated to each first transfer node.

[0072] For example, when monitoring metrics, we can first calculate idle time. Specifically, for any transit node, it receives tasks assigned by the scheduling node at intervals. The transit node accepts tasks assigned by the scheduling node, processes them, and waits for the next task to arrive. During this time, the scheduling node may repeatedly query the node's idleness. Because the current task is not yet completed, the scheduling node will not assign it a new task. Assume that the time of receiving a task is t1, the time of completing the task is t2, and the time of accepting the next task is t3. t2 is between t1 and t3. During the period from t1 to t3, its idle time Tidle = t3 - t2.

[0073] Then calculate the average processing rate, and set μ to be the amount of data processed per unit time by the transit node, i.e., the average processing rate. Furthermore, different types of data typically use different units of measurement. For example, when processing source server logs through a transit node in the embodiment of the present application, the number of log entries can be used as the unit of measurement, and the calculation formula is:

[0074]

[0075] Where p is the total amount of data processed by the transit node during the statistical time, Tidle is the total idle time of the transit node during the statistical time (i.e., the time period during which no data is actually processed), and T is the statistical time.

[0076] For example, the scheduling node obtains tasks from each source server and assigns them weightedly using μ as the weight. Assume that the scheduling node obtains n pieces of data from each source server at a time, and the μ values ​​of each processing node are μ1, μ2, ..., μn-1 respectively.

[0077] The amount of data allocated to transfer node k is:

[0078]

[0079] Then, the scheduling node may allocate the data to be synchronized to the destination server to each data processing transfer node according to the calculated amount of data allocated to each data processing transfer node.

[0080] In summary, dynamic load balancing based on statistical indicators can distribute synchronization tasks according to the node load capacity and reduce the load of each node as much as possible.

[0081] In order to quickly and accurately synchronize multiple target data to the destination server, in the data synchronization method provided in the embodiment of the present application, multiple target data can also be synchronized to the destination server through the following steps: obtaining the amount of data to be synchronized to the destination server from each source server per unit time; calculating the average queue length of each first transfer node based on the amount of data to be synchronized to the destination server from each source server per unit time, wherein the average queue length is used to represent the accumulation of multiple target data in the first time period; determining the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure of each first transfer node in the second time period, and the length of the second time period is less than the length of the first time period; based on the average queue length and real-time queue length of each first transfer node, sending a reminder message to the target object, wherein the reminder message is used to remind the target object to adjust the strategy for synchronizing multiple target data to the destination server.

[0082] For example, the scheduling node can record the following indicators: λ, which is the amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate, and its measurement unit is the same as μ. The calculation formula for λ is:

[0083]

[0084] Where T is the statistical time, and q is the total amount of data that needs to be synchronized by all source servers within the statistical time.

[0085] The scheduling node can then monitor the current load and calculate relevant metrics, providing them to users. These metrics include real-time queue length (L1) and average queue length (L2). Real-time queue length (L1) is the amount of data that has failed to be allocated, as recorded by the scheduling node. Real-time queue length reflects the current load pressure on the transit node within a short period of time.

[0086] The average queue length L2 reflects the data accumulation over a long period of time and whether the transfer node settings are reasonable. If the value is large, it is recommended to add transfer nodes. The calculation formula is as follows:

[0087]

[0088] In the formula, λ is the amount of data that all source servers need to synchronize per unit time, μ is the total amount of data processed by the transit nodes per unit time, that is,

[0089] Through the above solution, when synchronizing data in the source server to the destination server, the user can be reminded in time to adjust the data synchronization method, thereby improving the stability and efficiency of data synchronization.

[0090] In order to quickly and accurately handle faults of transit nodes, in the data synchronization method provided in the embodiment of the present application, the transit node can also be handled through the following steps: determine whether the second transit node fails; if the second transit node fails, determine a fourth transit node from multiple first transit nodes, wherein the fourth transit node is used to replace the second transit node to schedule transit nodes among multiple first transit nodes except the fourth transit node.

[0091] For example, in addition to achieving dynamic load balancing based on a queuing model, the device designed in the embodiment of the present application can also have a fault replacement function.

[0092] That is, when any transit node in the device fails, the device can quickly replace the failure.

[0093] If the current scheduling node fails, the device performs the following process:

[0094] (1) Elect a new scheduling node in order of number. For example, if the scheduling node numbered 1 fails, the node numbered 2 will be the new scheduling node. If the scheduling node numbered 2 fails, the node numbered 3 will be the new scheduling node.

[0095] (2) Re-execute the initialization deployment process, that is, redeploy the data synchronization device and use the new transfer node as the scheduling node;

[0096] (3) Re-execute the indicator monitoring process, that is, after initialization, the system runs and starts observation. Each processing node records μ, which is the amount of data processed by the transfer node per unit time, that is, the average processing rate, and the scheduling node records λ, which is the amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate;

[0097] (4) Re-execute the dynamic adjustment process, that is, the scheduling node dynamically adjusts the data allocated to each source server according to the indicator changes monitored during the indicator monitoring process.

[0098] To sum up, the scheduling node and the statistical node have exactly the same structure and can replace each other. If either fails, they can be quickly replaced.

[0099] In order to quickly and accurately handle faults of transit nodes, in the data synchronization method provided in the embodiment of the present application, the transit node can also be handled through the following steps: determine whether there is a faulty node among the multiple first transit nodes; if there is a faulty node among the multiple first transit nodes, determine multiple third transit nodes other than the faulty node from the multiple first transit nodes, wherein the third transit nodes are used to process multiple target data instead of the first transit nodes; obtain the efficiency of each third transit node in processing data; and allocate each target data to each third transit node based on the number of multiple target data and the efficiency of each third transit node in processing data.

[0100] For example, if one or more processing nodes fail, the device performs the following process:

[0101] The scheduling node records the faulty processing node, continues to execute the above indicator monitoring process, and updates the above dynamic adjustment process as follows:

[0102] The amount of data allocated to a certain transit node k (a transit node is a non-faulty node) is changed to:

[0103]

[0104] Where b represents the number of faulty nodes.

[0105] In addition, whether a scheduling node or a processing node fails, the real-time queue length L1 and the average queue length L2 may increase after a node failure occurs. At this time, the user can increase the number of transfer nodes and add new transfer nodes to the device. The newly added device automatically acts as a processing node. The joining process is as follows:

[0106] (1) Re-execute the initialization deployment process, that is, redeploy the data synchronization device and use the new transfer node as the scheduling node or data processing transfer node;

[0107] (2) Re-execute the indicator monitoring process, that is, after initialization, the system runs and starts observation. Each processing node records μ, which is the amount of data processed by the transfer node per unit time, that is, the average processing rate, and the scheduling node records λ, which is the amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate;

[0108] (3) Re-execute the dynamic adjustment process, that is, the scheduling node dynamically adjusts the data allocated to each source server according to the indicator changes monitored during the indicator monitoring process, and the n value in the above formula will include the newly added transfer node.

[0109] In summary, the scheduling node and the statistical node can replace each other. If either fails, they can be quickly replaced.

[0110] For example, Figure 6 is a flowchart of an optional data synchronization method provided according to an embodiment of the present application, such as Figure 6 As shown, the optional data synchronization method includes the following steps:

[0111] 1. Deployment device

[0112] The embodiment of the present application is deployed as follows, wherein the number of destination servers can be multiple. The embodiment of the present application does not focus on the binding relationship between the destination server and each transit node, and therefore regards the destination server as one.

[0113] Assume that there are m source servers and n transit nodes. The transit nodes synchronize data from the source servers, process it, and send it to the destination server.

[0114] Among all the transit nodes, one of them is regarded as a scheduling node, which is responsible for scheduling all transit nodes. The other nodes are responsible for data processing, also known as data processing nodes.

[0115] The internal structure of any transit node consists of a network module and a multi-process data processing module. The former is responsible for sending and receiving data, while the latter is responsible for data processing. Scheduling nodes are no different from data processing nodes; they are only selected during actual runtime to schedule other nodes. If necessary, any transit node can serve as a scheduling node.

[0116] In addition, the transit nodes are automatically numbered according to the time they join the system, and the transit node numbered 1 is automatically elected as the scheduling node. The scheduling node is responsible for node scheduling and task allocation, but is not responsible for data processing.

[0117] When the data to be processed occupies a large amount of bandwidth, but the scheduling node's transmission capacity is limited, the scheduling node assigns tasks by obtaining task-related information, such as the data number, from the source server without actually synchronizing the data. When data synchronization needs to be assigned to other transit nodes, a message is sent to inform the transit node of the task's corresponding source server, data number, and quantity, and the transit node then retrieves the data from the source server on its own.

[0118] When the data to be processed doesn't require significant bandwidth and the scheduling node has sufficient transmission capacity, the scheduling node dispatches tasks by obtaining task-related information and the data itself from the source server. When data synchronization needs to be assigned to other relay nodes, a message is sent to forward the data to them, eliminating the need for these other relay nodes to establish a connection with the source server.

[0119] During the initial system operation, a relatively simple round-robin load balancing method is used. The process is as follows: the dispatching node retrieves tasks from various source servers, traversing other relay nodes each time and allocating a fixed amount of tasks to each relay node. Before allocating tasks, the dispatching node checks whether the target relay node is idle. If so, the task is assigned to that relay node. If not, the dispatching node is queried for the next available relay node.

[0120] 2. Indicator monitoring

[0121] (1) Calculation of idle time

[0122] For any transit node, it accepts tasks assigned by the scheduling node at intervals.

[0123] A transit node accepts a task assigned by a scheduling node, processes it, and waits for the next task to arrive. During this time, the scheduling node may repeatedly query the node's availability. Because the current task is not yet completed, the scheduling node will not assign it a new task. Assume that the time a task is accepted is t1, the time the task is completed is t2, and the time the next task is accepted is t3. t2 is between t1 and t3. During the period from t1 to t3, the idle time Tidle = t3 - t2.

[0124] (2) Calculation of average processing rate and average arrival rate

[0125] After the initialization in Section 1 is completed, the system is running and observation begins. Each processing node records the following indicators:

[0126] μ: The amount of data processed per unit time by the transit node, i.e., the average processing rate. Different types of data usually use different units of measurement. For example, when processing source server logs through the transit node in the embodiment of the present application, the number of log entries can be used as the unit of measurement. Its calculation formula is:

[0127]

[0128] Where p is the total amount of data processed by the transit node during the statistical time, Tidle is the total idle time of the transit node during the statistical time (i.e., the time period during which no data is actually processed), and T is the statistical time.

[0129] The scheduling node records the following metrics:

[0130] λ: The amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate, and its unit of measurement is the same as μ. The calculation formula for λ is:

[0131]

[0132] Where T is the statistical time, and q is the total amount of data that needs to be synchronized by all source servers within the statistical time.

[0133] 3. Dynamic Adjustment

[0134] The purpose of dynamic adjustments based on detection indicators is to allocate tasks according to node load capacity, reducing the burden on each node, while also reducing the average data processing time and avoiding data retention. When the amount of data the system needs to process is too large and the individual transfer nodes are unable to handle it, the scheduling node will issue an early warning.

[0135] If the polling method in Section 1 is used continuously, it is very likely that a node will be overloaded for a long time. Therefore, dynamic adjustments need to be made according to the monitoring indicators in Section 2.

[0136] The scheduling node obtains tasks from each source server and assigns them weightedly using μ as the weight. Assume that the scheduling node obtains n pieces of data from each source server at a time, and the μ values ​​of each processing node are μ1, μ2, ..., μn-1 respectively.

[0137] The amount of data allocated to transfer node k is:

[0138]

[0139] The transit node counts μ and λ at a certain period. When μ and λ show significant difference after t test, the values ​​of the two are updated.

[0140] The dispatching node monitors the current load and calculates relevant metrics, providing them to users. These metrics include real-time queue length (L1) and average queue length (L2). Real-time queue length (L1) is the amount of data that has not been allocated, as recorded by the dispatching node. Real-time queue length reflects the current load pressure on the transit node within a short period of time.

[0141] The average queue length L2 reflects the data accumulation over a long period of time and whether the transfer node settings are reasonable. If the value is large, it is recommended to add transfer nodes. The calculation formula is as follows:

[0142]

[0143] In the formula, λ is the amount of data that all source servers need to synchronize per unit time.

[0144] μ: The total amount of data processed per unit time by the transfer node, that is

[0145] 4. Fault replacement

[0146] In addition to achieving dynamic load balancing based on a queuing model, the device designed in the embodiment of the present application also has a fault replacement function.

[0147] That is, when any transit node in the device fails, the device can quickly replace the failure.

[0148] If the current scheduling node fails, the device performs the following process:

[0149] (1) A new scheduling node is elected in order of number. For example, if the scheduling node numbered 1 fails, the node numbered 2 becomes the new scheduling node. If the scheduling node numbered 2 fails, the node numbered 3 becomes the new scheduling node.

[0150] (2) Re-execute the initialization process described in Section 1.

[0151] (3) Re-execute the indicator monitoring process described in Section 2.

[0152] (4) Re-execute the dynamic adjustment process described in Section 3.

[0153] If one or more processing nodes fail, the device performs the following process:

[0154] The scheduling node records the faulty processing node and continues to execute the indicator monitoring process described in Section 2. It updates the dynamic adjustment process described in Section 3 as follows:

[0155] The amount of data allocated to a certain transit node k (a transit node is a non-faulty node) is changed to:

[0156]

[0157] Where b represents the number of faulty nodes.

[0158] Regardless of whether a scheduling node or a processing node fails, the real-time queue length L1 and the average queue length L2 may increase after a node failure occurs. At this time, the user can increase the number of transfer nodes and add new transfer nodes to the device. The newly added device automatically acts as a processing node. The joining process is as follows:

[0159] (1) Re-execute the initialization process described in Section 1.

[0160] (2) Re-execute the indicator monitoring process described in Section 2.

[0161] (3) Re-execute the dynamic adjustment process described in Section 3. The value of n in the above formula will include the newly added transit nodes.

[0162] For example, in the audit and control of network security, security vendors often deploy proxy software in the internal network to synchronize users from the authentication server deployed in the cluster. The proxy software forwards the data to the network security device to help the network security device complete the real-name audit and control of traffic. In network security audits, the logs of different network security devices are usually different. In order to aggregate the logs to the log platform for unified management, a transit node can be set up to uniformly process and format the logs. In a small network, the cluster-deployed server 1 stores resources accessible to users, and server 2 is used to reverse proxy user requests and provide resources to users. A transit node is set between server 1 and server 2 to perform data decoding, classification, formatting and other processing operations.

[0163] Therefore, through the method provided in the embodiments of this application, multiple transit nodes can be used to dynamically connect to multiple source servers in a cluster environment, processing and forwarding data to the destination server. The binding relationship between the transit nodes and the source servers is dynamically adjusted based on the processing efficiency of the transit nodes and the data volume of the source servers, thereby improving the efficiency and stability of data synchronization.

[0164] In summary, the data synchronization method provided in the embodiment of the present application obtains multiple target data and multiple first transfer nodes through the second transfer node, wherein each target data is data to be synchronized to the destination server from multiple source servers in a cluster environment; determines the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; allocates each target data to each first transfer node through the second transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, which solves the problem in the related art of using a method of static binding of the source server in the cluster environment and the transfer node for processing data to synchronize the data of the source server in the cluster environment to the destination server, resulting in poor data synchronization effect. The second transfer node allocates each target data to each first transfer node based on the determined number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

[0165] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0166] Example 2

[0167] The present application also provides a data synchronization device. It should be noted that the data synchronization device of the present application can be used to execute the data synchronization method provided in the present application. The data synchronization device provided in the present application is introduced below.

[0168] Figure 7 Schematic diagram of a data synchronization device according to an embodiment of the present application. Figure 7 As shown, the device is applied in a data synchronization system, and the data synchronization system includes at least: multiple first transfer nodes and second transfer nodes, the first transfer nodes are used to process data, and the second transfer nodes are used to schedule multiple first transfer nodes. If there is a faulty node among the multiple first transfer nodes, the transfer nodes other than the faulty node in the first transfer nodes are used to replace the faulty node. If the second transfer node fails, any one of the multiple first transfer nodes is used to replace the second transfer node, and the device includes: a first acquisition module 701, a first determination module 702 and a first processing module 703.

[0169] Specifically, the first acquisition module 701 is configured to acquire multiple target data and multiple first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from multiple source servers to a destination server in a cluster environment;

[0170] A first determining module 702 is configured to determine, through the second transfer node, the number of target data and the efficiency of each first transfer node in processing data;

[0171] The first processing module 703 is used to allocate each target data to each first transfer node through the second transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronize the processed multiple target data to the destination server.

[0172] In summary, in the data synchronization device provided by the embodiment of the present application, the first acquisition module 701 acquires multiple target data and multiple first transfer nodes through the second transfer node, wherein each target data is data to be synchronized to the destination server from multiple source servers in the cluster environment; the first determination module 702 determines the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; the first processing module 703 allocates each target data to each first transfer node through the second transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, which solves the problem in the related art of using the method of static binding of the source server in the cluster environment and the transfer node for processing data to synchronize the data of the source server in the cluster environment to the destination server, resulting in poor data synchronization effect. The second transfer node allocates each target data to each first transfer node based on the determined number of multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

[0173] Optionally, in the data synchronization device provided in the embodiment of the present application, the first processing module includes: a first calculation unit, used to calculate the amount of data allocated to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data; a first allocation unit, used to allocate each target data to each first transfer node based on the amount of data allocated to each first transfer node.

[0174] Optionally, in the data synchronization device provided in the embodiment of the present application, the device also includes: a second determination module, used to determine multiple target transfer nodes before determining the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the multiple target transfer nodes include at least multiple first transfer nodes; a third determination module, used to determine the second transfer node from the multiple target transfer nodes, so as to determine the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes other than the multiple first transfer nodes.

[0175] Optionally, in the data synchronization device provided in the embodiment of the present application, the device also includes: a first judgment module, used to determine whether there is a faulty node among the multiple first transfer nodes after each target data is allocated to each first transfer node through the second transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data; a fourth determination module, used to determine multiple third transfer nodes other than the faulty node from the multiple first transfer nodes if there is a faulty node among the multiple first transfer nodes, wherein the third transfer node is used to process the multiple target data instead of the first transfer node; a second acquisition module, used to acquire the efficiency of each third transfer node in processing data; a first allocation module, used to allocate each target data to each third transfer node based on the number of multiple target data and the efficiency of each third transfer node in processing data.

[0176] Optionally, in the data synchronization device provided in the embodiment of the present application, the device also includes: a second judgment module, used to determine whether the second transfer node fails after determining the second transfer node from multiple target transfer nodes; a fifth determination module, used to determine a fourth transfer node from multiple first transfer nodes if the second transfer node fails, wherein the fourth transfer node is used to replace the second transfer node to schedule transfer nodes among multiple first transfer nodes except the fourth transfer node.

[0177] Optionally, in the data synchronization device provided in the embodiment of the present application, the first processing module includes: a first judgment unit, used to judge whether the data volume of multiple target data is greater than the preset data volume; a second allocation unit, used to allocate each target data to each first transfer node through the second transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data if the data volume of the multiple target data is greater than the preset data volume, so that the first transfer node obtains multiple target data from multiple source servers and processes the multiple target data; a first acquisition unit, used to obtain multiple target data from multiple source servers through the second transfer node if the data volume of the multiple target data is not greater than the preset data volume; a third allocation unit, used to allocate each target data obtained from multiple source servers to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data.

[0178] Optionally, in the data synchronization device provided in the embodiment of the present application, the third determination module includes: a second acquisition unit, used to obtain the numbering information corresponding to each target transit node; a first determination unit, used to determine the transit node with the smallest number among multiple target transit nodes based on the numbering information corresponding to each target transit node; and a second determination unit, used to use the transit node with the smallest number among multiple target transit nodes as the second transit node.

[0179] Optionally, in the data synchronization device provided in the embodiment of the present application, the device also includes: a third acquisition module, used to obtain the amount of data to be synchronized to the destination server from each source server per unit time; a first calculation module, used to calculate the average queue length of each first transfer node based on the amount of data to be synchronized to the destination server from each source server per unit time, wherein the average queue length is used to represent the accumulation of multiple target data in the first time period; a sixth determination module, used to determine the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure of each first transfer node in the second time period, and the length of the second time period is less than the length of the first time period; a first sending module, used to send a reminder message to the target object based on the average queue length and real-time queue length of each first transfer node, wherein the reminder message is used to remind the target object to adjust the strategy of synchronizing multiple target data to the destination server.

[0180] Optionally, in the data synchronization device provided in the embodiment of the present application, the device also includes: a third judgment module, used to determine whether the fifth transfer node among the multiple first transfer nodes is in an idle state after obtaining multiple target data and multiple first transfer nodes; a second allocation module, used to allocate the first data among the multiple target data to the fifth transfer node if the fifth transfer node among the multiple first transfer nodes is in an idle state, so that the fifth transfer node processes the first data, obtains the processed first data, and synchronizes the processed first data to the destination server; a fourth judgment module, used to determine whether the sixth transfer node among the multiple first transfer nodes is in an idle state if the fifth transfer node among the multiple first transfer nodes is not in an idle state; a third allocation module, used to allocate the first data among the multiple target data to the sixth transfer node if the sixth transfer node among the multiple first transfer nodes is in an idle state, so that the sixth transfer node processes the first data, obtains the processed first data, and synchronizes the processed first data to the destination server.

[0181] The data synchronization device includes a processor and a memory. The first acquisition module 701, the first determination module 702 and the first processing module 703 are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0182] The processor contains a kernel, which retrieves the corresponding program unit from the memory. You can set one or more kernels, and adjust the kernel parameters to improve data synchronization.

[0183] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0184] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data synchronization method is implemented.

[0185] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data synchronization method is executed when the program is running.

[0186] An embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: obtaining multiple target data and the multiple first transfer nodes through a second transfer node, wherein each target data is data to be synchronized to a destination server from multiple source servers in a cluster environment; determining the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the multiple processed target data to the destination server.

[0187] When the processor executes the program, the following steps are also implemented: each target data is allocated to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, including: the amount of data allocated to each first transfer node is calculated based on the number of the multiple target data and the efficiency of each first transfer node in processing data; each target data is allocated to each first transfer node based on the amount of data allocated to each first transfer node.

[0188] When the processor executes the program, the following steps are also implemented: before determining the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, the method also includes: determining multiple target transfer nodes, wherein the multiple target transfer nodes include at least the multiple first transfer nodes; determining the second transfer node from the multiple target transfer nodes to determine the number of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes other than the multiple first transfer nodes.

[0189] When the processor executes the program, the following steps are also implemented: after each target data is allocated to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, the method also includes: judging whether there is a faulty node among the multiple first transfer nodes; if there is a faulty node among the multiple first transfer nodes, determining multiple third transfer nodes other than the faulty node from the multiple first transfer nodes, wherein the third transfer node is used to process the multiple target data instead of the first transfer node; obtaining the efficiency of each third transfer node in processing data; and allocating each target data to each third transfer node based on the number of the multiple target data and the efficiency of each third transfer node in processing data.

[0190] When the processor executes the program, the following steps are also implemented: after determining the second transfer node from the multiple target transfer nodes, the method also includes: judging whether the second transfer node fails; if the second transfer node fails, determining a fourth transfer node from the multiple first transfer nodes, wherein the fourth transfer node is used to replace the second transfer node to schedule the transfer nodes among the multiple first transfer nodes except the fourth transfer node.

[0191] When the processor executes the program, the following steps are also implemented: allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, including: judging whether the data volume of the multiple target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, allocating each target data to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, so that the first transfer node obtains the multiple target data from the multiple source servers and processes the multiple target data; if the data volume of the multiple target data is not greater than the preset data volume, obtaining the multiple target data from the multiple source servers through the second transfer node; and allocating each target data obtained from the multiple source servers to each first transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data.

[0192] When the processor executes the program, the following steps are also implemented: determining the second transfer node from the multiple target transfer nodes includes: obtaining the numbering information corresponding to each target transfer node; based on the numbering information corresponding to each target transfer node, determining the transfer node with the smallest number among the multiple target transfer nodes; and using the transfer node with the smallest number among the multiple target transfer nodes as the second transfer node.

[0193] When the processor executes the program, the following steps are also implemented: the method also includes: obtaining the amount of data to be synchronized to the destination server from each source server per unit time; calculating the average queue length of each first transfer node based on the amount of data to be synchronized to the destination server from each source server per unit time, wherein the average queue length is used to represent the accumulation of the multiple target data in the first time period; determining the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure of each first transfer node in the second time period, and the length of the second time period is less than the length of the first time period; based on the average queue length and real-time queue length of each first transfer node, sending a reminder message to the target object, wherein the reminder message is used to remind the target object to adjust the strategy of synchronizing the multiple target data to the destination server.

[0194] When the processor executes the program, the following steps are also implemented: after obtaining multiple target data and multiple first transfer nodes, the method also includes: judging whether the fifth transfer node among the multiple first transfer nodes is in an idle state; if the fifth transfer node among the multiple first transfer nodes is in an idle state, the first data among the multiple target data is allocated to the fifth transfer node, so that the fifth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server; if the fifth transfer node among the multiple first transfer nodes is not in an idle state, judging whether the sixth transfer node among the multiple first transfer nodes is in an idle state; if the sixth transfer node among the multiple first transfer nodes is in an idle state, the first data among the multiple target data is allocated to the sixth transfer node, so that the sixth transfer node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server.

[0195] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0196] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0200] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0201] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0202] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0203] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0204] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data synchronization method, characterized in that: The method is applied in a data synchronization system, wherein the data synchronization system includes at least a plurality of first transfer nodes and a second transfer node, wherein the first transfer node is used to process data, and the second transfer node is used to schedule the plurality of first transfer nodes; if there is a faulty node among the plurality of first transfer nodes, a transfer node other than the faulty node among the first transfer nodes is used to replace the faulty node; If the second transfer node fails, any one of the plurality of first transfer nodes is used to replace the second transfer node; the synchronization method specifically includes: Acquire multiple target data and the multiple first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from multiple source servers to a destination server in a cluster environment; Determining, by the second transfer node, the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data; Allocating each target data to each first transfer node by the second transfer node according to the quantity of the multiple target data and the efficiency of each first transfer node in processing data, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizing the processed multiple target data to the destination server; The allocating each target data to each first transfer node by the second transfer node according to the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data includes: Determining whether the data volume of the plurality of target data is greater than a preset data volume; If the data amount of the multiple target data is greater than the preset data amount, each target data is allocated to each first transfer node by the second transfer node according to the amount of the multiple target data and the efficiency of each first transfer node in processing data, so that the first transfer node obtains the multiple target data from the multiple source servers and processes the multiple target data; If the data volume of the multiple target data is not greater than the preset data volume, obtaining the multiple target data from the multiple source servers through the second transfer node; Each target data obtained from the multiple source servers is allocated to each first transfer node according to the quantity of the multiple target data and the efficiency of each first transfer node in processing data.

2. The method according to claim 1, characterized in that Allocating each target data to each first transfer node by the second transfer node according to the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data includes: Calculating the amount of data allocated to each first transfer node according to the amount of the plurality of target data and the efficiency of each first transfer node in processing data; Each target data is allocated to each first transit node based on the amount of data allocated to each first transit node.

3. The method according to claim 1, characterized in that Before determining, by the second transfer node, the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data, the method further includes: Determining a plurality of target transfer nodes, wherein the plurality of target transfer nodes include at least the plurality of first transfer nodes; Determine the second transfer node from the multiple target transfer nodes to determine the quantity of the multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, wherein the second transfer node is a transfer node among the multiple target transfer nodes other than the multiple first transfer nodes.

4. The method according to claim 1, wherein After allocating each target data to each first transfer node by the second transfer node according to the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data, the method further includes: Determining whether there is a faulty node among the multiple first transfer nodes; If there is a faulty node among the plurality of first transfer nodes, determining a plurality of third transfer nodes excluding the faulty node from the plurality of first transfer nodes, wherein the third transfer nodes are used to replace the first transfer nodes to process the plurality of target data; Obtain the data processing efficiency of each third transfer node; Each target data is allocated to each third transit node according to the quantity of the plurality of target data and the data processing efficiency of each third transit node.

5. The method according to claim 3, characterized in that After determining the second transfer node from the multiple target transfer nodes, the method further includes: Determining whether the second transit node fails; If the second transfer node fails, a fourth transfer node is determined from the multiple first transfer nodes, wherein the fourth transfer node is used to replace the second transfer node to schedule the transfer nodes in the multiple first transfer nodes except the fourth transfer node.

6. The method according to claim 3, characterized in that Determining the second transfer node from the multiple target transfer nodes includes: Get the number information corresponding to each target transfer node; Determine the transfer node with the smallest number among the multiple target transfer nodes based on the number information corresponding to each target transfer node; The transfer node with the smallest number among the multiple target transfer nodes is used as the second transfer node.

7. A data synchronization device, characterized in that: The device is applied in a data synchronization system, which includes at least: a plurality of first transfer nodes and a second transfer node, wherein the first transfer node is used to process data, and the second transfer node is used to schedule the plurality of first transfer nodes; if a faulty node exists among the plurality of first transfer nodes, a transfer node other than the faulty node among the first transfer nodes is used to replace the faulty node; if the second transfer node fails, any one of the plurality of first transfer nodes is used to replace the second transfer node, including: A first acquisition module is configured to acquire, through the second transfer node, a plurality of target data and the plurality of first transfer nodes, wherein each target data is data to be synchronized from a plurality of source servers to a destination server in a cluster environment; A first determining module, configured to determine, through the second transfer node, the quantity of the plurality of target data and the efficiency of each first transfer node in processing the data; a first processing module, configured to distribute each target data to each first transfer node through the second transfer node according to the quantity of the multiple target data and the data processing efficiency of each first transfer node, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronize the processed multiple target data to the destination server; In which, the device is also used to determine whether the data volume of the multiple target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, each target data is allocated to each first transfer node through the second transfer node based on the number of the multiple target data and the efficiency of each first transfer node in processing data, so that the first transfer node obtains the multiple target data from the multiple source servers and processes the multiple target data; if the data volume of the multiple target data is not greater than the preset data volume, the multiple target data is obtained from the multiple source servers through the second transfer node; based on the number of the multiple target data and the efficiency of each first transfer node in processing data, each target data obtained from the multiple source servers is allocated to each first transfer node.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, wherein the program executes the data synchronization method according to any one of claims 1 to 6.

9. A processor for a data synchronization method, characterized in that: The processor is configured to run a program, wherein the program, when running, executes the data synchronization method according to any one of claims 1 to 6.

Images