Data replication method, device and equipment and computer readable storage medium

By determining the initial replicated target data node in the cluster and using it for exponential replication, the problem of low file replication efficiency in the cluster is solved, and the replication time and efficiency improvement are achieved.

CN119003246APending Publication Date: 2024-11-22JINAN INSPUR DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411203299.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The efficiency of copying files to each node in the cluster is low. As the cluster scale expands, the transmission time increases linearly, resulting in a decrease in replication efficiency.

Method used

By determining the initial replicated target data node, the target data is copied to a preset number of targetless data nodes, and the replicated target data node continues to replicate to other targetless data nodes until all nodes are replicated, using exponential growth improve replication efficiency.

Benefits of technology

By fully utilizing the replicated target data node as a replication source, the replication efficiency is significantly improved, the replication time is shortened, and the efficiency of file replication is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003246A_ABST
    Figure CN119003246A_ABST
Patent Text Reader

Abstract

The invention discloses a data replication method, device and equipment and a computer readable storage medium, which are applied to the technical field of data processing, and the method comprises the following steps: determining an initial replicated target data node; the initial copied target data node is a node with target data, and the target data is data needing to be copied by each node; copying the target data to a preset number of non-target data nodes by using the initial copied target data nodes, and repeatedly executing the process of copying the target data to the non-target data nodes by using the copied target data nodes until all the nodes copy the target data; the copied target data nodes are nodes with copied target data in the current cluster. According to the method, the copied target data node obtained each time is used as a copying source, the file is transmitted to other nodes without target data, and along with the continuous increase of the copied target data nodes, exponentially increased copying is realized, the copying time is shortened, and the file copying efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data replication method, device, equipment and computer-readable storage medium. Background Art

[0002] The main purpose of using clusters is to improve the stability and availability of the system and the data processing and service capabilities of the network center. At this time, it is required that the program data in the server cluster is completely consistent. When copying file data to each node, the usual practice is to obtain the cluster node list and upload files to each node one by one. This method of uploading to one node and then uploading to another node has obvious disadvantages. As the size of the cluster increases, the transmission time will also increase linearly, making the copying efficiency slower.

[0003] It can be seen that how to improve the efficiency of copying files to each node in a cluster is a problem that those skilled in the art need to solve. Summary of the invention

[0004] In view of this, an object of the embodiments of the present invention is to provide a data replication method, apparatus, device and computer-readable storage medium, which can solve the technical problem of low efficiency of replicating target data by nodes in a cluster.

[0005] In order to solve the above technical problems, the present invention provides a data replication method, comprising:

[0006] Determine an initial replicated target data node; wherein the initial replicated target data node is a node that has target data, and the target data is data that needs to be replicated for each node;

[0007] Using the initial replicated target data node, the target data is replicated to a preset number of first batch nodes without target data, to obtain a preset number of first batch replicated target data nodes; wherein the preset number is a number determined according to node performance;

[0008] Using the initial replicated target data nodes and the first batch of replicated target data nodes, the target data is respectively replicated to the corresponding second batch of non-target data nodes to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of non-target data nodes;

[0009] Repeat the process of using replicated target data nodes to replicate the target data to nodes without target data until all nodes have replicated the target data; wherein the replicated target data nodes are nodes in the current cluster that have replicated the target data.

[0010] In some embodiments, using the initial replicated target data node to replicate the target data to a preset number of first batch nodes without target data to obtain a preset number of first batch replicated target data nodes includes:

[0011] Number the nodes according to the node address or node host name to obtain the serial number corresponding to each node;

[0012] The target data is copied to a preset number of the first batch of target-free data nodes using the initial copied target data nodes according to the sequence number corresponding to each node, so as to obtain a preset number of the first batch of copied target data nodes.

[0013] In some embodiments, the preset number is x, the number of rounds of replication is n, the base X=x+1, and the serial number of the replicated target data node is m;

[0014] When the nth round of replication is completed, the number of nodes that have replicated the target data node is , when the target data is currently being copied, the copy mapping relationship is determined as follows: the sequence number of the node without target data corresponding to the copied target data node with sequence number m is , Y is an integer from 1 to x;

[0015] The target data on each replicated target data node is replicated to a corresponding non-target data node using the replication mapping relationship.

[0016] In some embodiments, using the initial replicated target data nodes and the first batch of replicated target data nodes to respectively replicate the target data to corresponding second batch of target-free data nodes to obtain the second batch of replicated target data nodes includes:

[0017] The target data is copied to the second batch of non-target data nodes respectively according to the initial replicated target data nodes and the first batch of replicated target data nodes by using an adjacent replication method to obtain the second batch of replicated target data nodes; wherein, the adjacent replication method is that the non-target data nodes corresponding to each replicated target data node are a preset number of adjacent nodes.

[0018] In some embodiments, the step of numbering the nodes according to the node addresses or node host names to obtain the sequence number corresponding to each node includes:

[0019] The nodes are numbered by using a sequence number processing command through a pipe symbol to obtain the sequence number corresponding to each node.

[0020] In some embodiments, during the copying process, the following is further included:

[0021] All replication threads are stored using a thread list; wherein the thread list includes replicated target data nodes, non-target data nodes, and replication results; the replication result is the result of whether the target data has been replicated; the replication thread is the process of replicating the target data from the replicated target data node to the non-target data node;

[0022] Record the copy result corresponding to the node in the thread list into a result dictionary; wherein the result dictionary uses the node name as a key and the copy result as a value;

[0023] Traversing the result dictionary, and determining the replication status corresponding to each replication thread according to the result dictionary; wherein the replication status includes replication success, replication in progress, and replication error;

[0024] Wherein, determining the replication status according to the result dictionary includes:

[0025] Determine whether the number of successful copies in the result dictionary is equal to the number of current copy threads;

[0026] When the number of successful copies in the result dictionary is equal to the number of current copy threads, it is determined that the copying is complete and the next round of copying is performed;

[0027] When the number of successful copies in the result dictionary is not equal to the number of current copy threads, determining whether the copy status corresponding to the remaining threads is a copy error;

[0028] When the replication status corresponding to the remaining threads is a replication error, determining a target data node corresponding to the replication error;

[0029] When the replication status corresponding to the remaining threads is not replication error, it is determined to wait for the replication to end.

[0030] In some embodiments, during the copying process, the following is further included:

[0031] Using the initial replicated target data node to store the completed replication node information, obtaining replication process information;

[0032] Determine whether to execute the next round of replication according to the replication progress information; wherein the replication progress information is information on whether the replication process is completed.

[0033] The present invention also provides a data replication device, comprising:

[0034] A node determination module is used to determine an initial replicated target data node; wherein the initial replicated target data node is a node that has target data, and the target data is data that needs to be replicated for each node;

[0035] A first replication module is used to replicate the target data to a preset number of first batch nodes without target data by using the initial replicated target data node, so as to obtain a preset number of first batch replicated target data nodes; wherein the preset number is a number determined according to node performance;

[0036] A second replication module is used to use the initial replicated target data node and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of non-target data nodes to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of non-target data nodes;

[0037] The third replication module is used to repeatedly execute the process of using the replicated target data node to replicate the target data to the node without target data until all nodes have replicated the target data; wherein the replicated target data node is a node in the current cluster that has replicated the target data.

[0038] The present invention also provides a data replication device, comprising:

[0039] Memory for storing computer programs;

[0040] A processor is used to execute the computer program to implement the steps of the above data replication method.

[0041] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned data replication method are implemented.

[0042] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the data replication method when executed by a processor.

[0043] To solve the above technical problems, an embodiment of the present invention provides a data replication method, which may include: determining an initial replicated target data node; wherein the initial replicated target data node is a node that already has target data, and the target data is the data that each node needs to replicate; using the initial replicated target data node to replicate the target data to a preset number of first batch of nodes without target data, to obtain a preset number of first batch of replicated target data nodes; wherein the preset number is a number determined according to node performance; using the initial replicated target data node and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of nodes without target data, to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of nodes without target data; repeatedly executing the process of replicating the target data to the nodes without target data using the replicated target data nodes until all nodes have replicated the target data; wherein the replicated target data nodes are nodes in the current cluster that have replicated the target data.

[0044] It can be seen from the above technical solution that, compared with the current method of obtaining a list of cluster nodes and uploading target data to each node one by one, the beneficial effect of the present invention is: making full use of the nodes that have copied the target data in each round, allowing them to serve as replication sources to transmit files to other nodes that do not have target data. Since the number of nodes that have copied target data is constantly increasing, exponentially growing replication is achieved, which shortens the replication time and improves the efficiency of file replication. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a data replication method provided by an embodiment of the present invention;

[0047] Figure 2 An example diagram of a first replication relationship provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a second replication relationship provided by an embodiment of the present invention;

[0049] Figure 4 An example flow chart of a data replication method provided by an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of a data replication device provided by an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a data replication device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0054] Some of the terms that appear in the description of the embodiments of the present invention may be interpreted as follows:

[0055] Shell: is a command interpreter that interprets the commands entered by the user and sends them to the kernel.

[0056] SSH: is a protocol used for secure remote login and other secure network services on an insecure network, which can execute commands remotely.

[0057] scp: A remote file transfer tool based on ssh.

[0058] Service cluster: A collection of multiple service instances that can work together to provide a specific function or service.

[0059] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0060] Next, a flow chart of a data replication method provided by an embodiment of the present invention is described in detail. Figure 1 A data replication method provided by an embodiment of the present invention may include:

[0061] S101, determining an initial replicated target data node; wherein the initial replicated target data node is a node that already has target data, and the target data is data that needs to be replicated for each node.

[0062] This embodiment does not limit the specific execution subject, and the execution subject of this embodiment can be each node, or other devices in the cluster. The initial replicated target data node in this embodiment is the node corresponding to the node that replicates the target data to the cluster for the first time. This embodiment does not limit the number of replicated target data nodes. For example, the number of replicated target data nodes in this embodiment can be 1; or the number of replicated target data nodes in this embodiment can be 2. This embodiment does not limit the specific target data. For example, the target data in this embodiment can be an upgrade package file; or the target data in this embodiment can also be a debugging program.

[0063] S102, using the initial replicated target data nodes to replicate the target data to a preset number of first batch of nodes without target data, to obtain a preset number of first batch of replicated target data nodes; wherein the preset number is a number determined according to node performance.

[0064] The preset number in this embodiment is the number of target-free data nodes that a single machine can copy to at the same time, in order to control the number of outward copies of a single existing target data node and control the bandwidth usage rate. This embodiment does not limit the specific preset number. For example, the preset number in this embodiment can be 2, or the preset number in this embodiment can be 3. This embodiment numbers the nodes, determines the sequence number of the nodes, and determines the target-free data node corresponding to each copied target data node according to the sequence number.

[0065] It should be further explained that, in order to improve the efficiency of replication, the above-mentioned numbering of nodes and determining the sequence number of nodes may include: obtaining the performance data of each node, determining the performance of the node according to the performance data; sorting the nodes from high to low according to the performance of each node, and determining the sequence number of the node. In this embodiment, determining the performance of the nodes in the cluster generally involves the following aspects: CPU (central processing unit) performance: The CPU is the core of computing resources, and its performance directly affects the processing capacity of the node. The CPU performance of the node can be evaluated by viewing indicators such as CPU model, number of cores, and frequency. Memory capacity and speed: Memory is essential for running applications and storing temporary data. The larger the memory capacity, the more tasks can be processed simultaneously; the faster the memory speed, the faster the data can be accessed. Storage performance: The performance of the storage device determines the speed of reading and writing data. Common storage performance indicators include disk type, rotation speed, cache size, etc. Network bandwidth: Network bandwidth determines the ability of nodes to communicate with each other and with external networks. High bandwidth means faster data exchange speed, which is very important for distributed tasks and large-scale data processing. I / O (input and output) performance: The speed of I / O operations (such as reading and writing files) also affects the performance of the node. High-speed I / O devices can improve data transmission efficiency. Operating system and software configuration: Different operating systems and software configurations may affect node performance. For example, the kernel version of the operating system, virtualization technology, network protocol stack, etc. may affect the performance of the node. This embodiment determines the node sequence number by numbering the nodes according to the node performance, and preferentially copies the target data to the node with good node performance. Since the performance of the node to be copied is relatively good, the efficiency of copying the target data can be further improved.

[0066] It should be further explained that, in order to improve the accuracy of replication, the above-mentioned use of the initial replicated target data node to replicate the target data to a preset number of the first batch of target-free data nodes to obtain a preset number of the first batch of replicated target data nodes may include:

[0067] S1021, number the nodes according to the node addresses or node host names to obtain a sequence number corresponding to each node.

[0068] S1022, using the initial replicated target data node to replicate the target data to a preset number of first batch nodes without target data according to the sequence number corresponding to each node, to obtain a preset number of first batch replicated target data nodes.

[0069] This embodiment can use a function to obtain a list containing node addresses and return a dictionary, where the key is the node address and the value is the corresponding serial number. The serial number starts from 1 and increases according to the order of the nodes in the list. This embodiment can also receive a list containing node host names and return a dictionary, where the key is the node host name and the value is the corresponding serial number. The serial number starts from 1 and increases according to the order of the nodes in the list. This embodiment determines the serial number corresponding to each node according to the node address or the node host name, so that in the process of copying, the node without target data is directly determined according to the node serial number, and the target data is copied in an orderly manner, thereby improving the accuracy of target data copying and preventing omissions.

[0070] It should be further explained that in order to improve the accuracy of replication, the above preset number is x, the number of replication rounds is n, the base X=x+1, and the serial number of the replicated target data node is m; when the nth round of replication, the number of nodes of the replicated target data node is , when the target data is currently being copied, the copy mapping relationship is determined as follows: the sequence number of the node without target data corresponding to the copied target data node with sequence number m is , Y is an integer from 1 to x; the target data on each replicated target data node is replicated to the corresponding non-target data node using the replication mapping relationship. The replication mapping relationship in this embodiment includes the relationship between each replicated target data node and the non-target data node; the mapping relationship when the target data is replicated using the replication mapping relationship is from sequence number 1 to , respectively copy to to . That is, the m number (i.e. ) node files are copied to Number, Number, Number.

[0071] For ease of understanding, for example, in this embodiment, when the target data is replicated for the first time using the initial replicated target data node, as the first round of replication, when the preset number is 2, the base number is 3. At this time, the replication mapping relationship is that the initial replicated target data node with sequence number 1 corresponds to the non-target data node with sequence number 2 and the non-target data node with sequence number 3. At this time, it means that the target data is replicated to the non-target data nodes with sequence number 2 and sequence number 3 using the initial replicated target data node with sequence number 1. Please refer to Figure 2 , Figure 2 This is an example diagram of the first replication relationship provided by the embodiment of the present invention, from Figure 2It can be seen that in the first round of replication, the number of replicated target data nodes is 1, in the second round of replication, the number of replicated target data nodes is 3, and in the third round of replication, the number of replicated target data nodes is 9. This embodiment provides a specific replication mapping relationship so that the corresponding non-target data nodes can be determined by simple calculation during the replication process, thereby improving the efficiency of copying the target data to the non-target data nodes. Accordingly, Table 1 is an example table of a replication relationship provided by an embodiment of the present invention. In the node data in Table 1, "Yes" indicates a node with target data, and "No" indicates a node without target data.

[0072] Table 1 Example table of a replication relationship

[0073]

[0074] It should be further explained that the above-mentioned use of the initial replicated target data nodes and the first batch of replicated target data nodes to copy the target data to the corresponding second batch of non-target data nodes respectively to obtain the second batch of replicated target data nodes may include: using an adjacent replication method to copy the target data to the second batch of non-target data nodes respectively according to the initial replicated target data nodes and the first batch of replicated target data nodes to obtain the second batch of replicated target data nodes; wherein the adjacent replication method is that the non-target data nodes corresponding to each replicated target data node are a preset number of adjacent nodes. The adjacent replication method in this embodiment is to use the replicated target data nodes to copy the target data to two adjacent non-target data nodes. For ease of understanding, please refer to Figure 3 , Figure 3 A schematic diagram of a second replication relationship provided by an embodiment of the present invention, from Figure 3 It can be seen that the node with sequence number 1 corresponds to the target data nodes 4 and 5, the node with sequence number 2 corresponds to the target data nodes 6 and 7, and the node with sequence number 3 corresponds to the target data nodes 8 and 9. This embodiment implements replication by treating two adjacent target data nodes as child nodes of a replicated target data node.

[0075] It should be further explained that, in order to improve the efficiency of numbering, the above-mentioned numbering of nodes according to node addresses or node host names to obtain the serial number corresponding to each node may include: using the serial number processing command to number the nodes through the pipe symbol to obtain the serial number corresponding to each node. This embodiment uses the serial number processing command to number the nodes through the pipe symbol, which can be implemented using the awk command, which is a text processing command. Assume that we have a file nodes.txt containing node addresses, one node address per line, this command will output the line number (i.e., the serial number) and the corresponding node address of each line.

[0076] S103, using the initial replicated target data nodes and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of non-target data nodes, to obtain the second batch of replicated target data nodes; wherein each node can replicate the target data to a preset number of non-target data nodes.

[0077] For ease of understanding, for example, the initial replicated target data node is 1, the preset number is 2, and the initial replicated target data node is used to replicate the target data to the preset number of first batch non-target data nodes, obtaining the preset number of first batch replicated target data nodes (2 first batch replicated target data nodes). Using 1 initial replicated target data node and 2 first batch replicated target data nodes as replicated target data nodes, 1 initial replicated target data node can replicate the target data to 2 non-target data nodes, and 2 first batch replicated target data nodes can replicate the target data node to 4 non-target data nodes. At this time, the number of the second batch replicated target data nodes is 6.

[0078] S104, repeatedly executing the process of using replicated target data nodes to replicate target data to nodes without target data until all nodes have replicated the target data; wherein the replicated target data nodes are nodes in the current cluster that have replicated the target data.

[0079] This embodiment repeatedly executes the process of using replicated target data nodes to copy target data to nodes without target data until all nodes have copied target data, which means that in each round, the target data is copied to nodes without target data using replicated target data nodes.

[0080] It should be further explained that in order to ensure that the target data is successfully copied to the node without target data, the copy process may also include:

[0081] S1041, using a thread list to store all replication threads; wherein the thread list includes replicated target data nodes, non-target data nodes and replication results; the replication result is the result of whether the target data has been replicated; the replication thread is the process of replicating the target data from the replicated target data node to the non-target data node.

[0082] S1042, recording the copy result corresponding to the node in the thread list into a result dictionary; wherein the result dictionary uses the node name as a key and the copy result as a value.

[0083] S1043, traverse the result dictionary, and determine the replication status corresponding to each replication thread according to the result dictionary; wherein the replication status includes replication success, replication in progress, and replication error.

[0084] The replication status is determined according to the result dictionary, including:

[0085] S1044, determining whether the number of successful copies in the result dictionary is equal to the number of current copy threads.

[0086] S1045, when the number of successfully copied in the result dictionary is equal to the number of current copy threads, it is determined that the copying is complete and the next round of copying is performed.

[0087] S1046: When the number of successful copies in the result dictionary is not equal to the number of current copy threads, determine whether the copy status corresponding to the remaining threads is a copy error.

[0088] S1047, when the replication status corresponding to the remaining threads is replication error, determine the target data node corresponding to the replication error.

[0089] S1048: When the copy status corresponding to the remaining threads is not a copy error, determine to wait for the copy to end.

[0090] This embodiment can timely discover and handle errors in the replication process by recording the replication results of each node, avoiding the situation where the entire replication process fails due to a single node failure. When the amount of data to be replicated is large, the replication speed can be increased by increasing the number of threads to meet the needs of different scales.

[0091] It should be further explained that in order to improve the accuracy of replication, the replication process may also include: using the information of the replication node stored in the initial replicated target data node to obtain the replication process information; determining whether to execute the next round of replication according to the replication process information; wherein the replication process information is information on whether the replication process is completed. The replication process information in this embodiment is to determine the replication process, such as a progress bar, or whether the replication is successful. This embodiment determines to execute the next round of replication when the replication process information indicates that the replication is completed, and determines to wait when the replication process information indicates that the replication is not yet completed. This embodiment can determine the node replication result through the replication process information, so as to quickly perform the next round of replication.

[0092] A data replication method improved by an embodiment of the present invention may include: S101, determining an initial replicated target data node; wherein the initial replicated target data node is a node that already has target data, and the target data is data that needs to be replicated for each node; S102, using the initial replicated target data node to replicate the target data to a preset number of first batch of nodes without target data, to obtain a preset number of first batch of replicated target data nodes; wherein the preset number is a number determined according to node performance; S103, using the initial replicated target data node and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of nodes without target data, to obtain a second batch of replicated target data nodes; wherein each node can replicate the target data to a preset number of nodes without target data; S104, repeatedly executing the process of using the replicated target data node to replicate the target data to the nodes without target data until all nodes have replicated the target data; wherein the replicated target data node is a node in the current cluster that has replicated the target data. The embodiment of the present invention makes full use of the nodes that have copied the target data in each round, and allows them to be used as the copy source to transfer files to other nodes without target data. Since the number of nodes that have copied the target data is constantly increasing, exponentially growing replication is achieved, the replication time is shortened, and the efficiency of file replication is improved; and, the embodiment determines the serial number corresponding to each node according to the node address or the node host name, so that the node without target data is directly determined according to the node serial number during the replication process, and the target data is copied, thereby improving the accuracy of target data replication and preventing repeated replication; and, by giving a specific replication mapping relationship, the corresponding node without target data can be determined by simple calculation during the replication process, thereby improving the efficiency of copying the target data to the node without target data; and, by recording the replication results of each node, the embodiment can timely discover and handle errors in the replication process, and avoid the situation where the entire replication process fails due to the failure of a single node. When the amount of data to be copied is large, the replication speed can be increased by increasing the number of threads to meet the needs of different scales; and, the embodiment can determine the replication result through the replication process information, so as to quickly perform the next round of replication.

[0093] In order to make the present invention easier to understand, please refer to Figure 4 , Figure 4 A flowchart of a data replication method provided by an embodiment of the present invention may specifically include:

[0094] S401, numbering the nodes using each node address or host name to obtain a sequence number of the node.

[0095] The application scenario of this embodiment can be a service cluster, which is a server group that manages and participates in workload management together. Two or more servers are connected through a high-speed network so that these servers work together to provide users with services such as applications, system resources and data. From the client's perspective, the cluster is like having only one server. The main purpose of using a cluster is to improve the stability, availability, and data processing and service capabilities of the system center. At this time, in most cases, it is required that the program data in the server cluster is completely consistent. In daily upgrades and maintenance, it is necessary to upload some upgrade packages, debugging programs, etc. to the cluster, and ensure that these files are consistent on all nodes in the cluster. Or the application scenario of this embodiment can also be CDN (Content Delivery Network). The present invention is not limited to copying target data within a cluster, and the method can also be used for synchronous sharing of large-scale data within a cluster. This embodiment can roughly divide the replication process into two parts: synchronization of node numbering and replication information within the cluster, and target data replication logic. In order to facilitate management, the cluster generally does not require secret login between nodes. If not configured, it can also be configured to facilitate copying. The following operations can use scp (Secure Copy, securely transfer files between the local system and the remote system) as a copy tool; or this embodiment can also use sftp (Secure File Transfer Protocol, secure file transfer protocol) as a copy tool. When numbering in this embodiment, the address or host name of each node can be added with a serial number starting from 1 through some cluster management tools or some other methods. This numbering step is very critical and is dependent on replication. (How to add serial numbers, we can use the awk command to add serial numbers to each host name through the pipe symbol, such as: awk 'NR {print NR":"$0";"}'), the format is as follows: 1 hostname1; 2 hostname2; ... This embodiment needs to copy the node serial number file and the target data replication logic to all nodes in the cluster for subsequent replication.

[0096] S402, determining the number of replication transmissions of the initial node and the preset node, sending the target data to the initial node, and obtaining the initial replicated target data node.

[0097] In this embodiment, the target data of each node (the node with copied target data) can be copied to the corresponding node without target data through scp, and the file can be copied to other nodes through scp.

[0098] S403, using the initial replicated target data nodes to replicate the target data to a first batch of nodes without target data based on a preset node replication transmission quantity and a node sequence number, to obtain a plurality of replicated target data nodes.

[0099] In this embodiment, each replicated target data node corresponds to a preset node replication transmission number of target-free data nodes. In this embodiment, the node of the replicated target data obtained by the previous step can be triggered by ssh remote command call, and the initial replicated target data node is used as a node with target data to copy the target data to other target-free data nodes. During the ssh process, it is necessary to determine which round of replication process it is in order to know which target-free data nodes need to deliver target data. The entire ssh is executed in the form of a background program so that all child nodes can synchronously start the next round of replication tasks. In this embodiment, the preset node replication transmission number corresponds to the preset number above. At this time, the time for n nodes to complete the replication is approximately log2(n), and 1 is the preset node replication transmission number. When the bandwidth or business allows each node to replicate to two or three or even more nodes at a time, the completion time will be shortened to log3(n), log4(n) or even lower.

[0100] S404: Determine the progress of multiple replication processes using the initial replicated target data node.

[0101] In order to synchronize the replication information, this embodiment starts multiple replications at the same time, waits for the results of multiple simultaneous replication processes, and avoids exceeding the limit of the replication process executed by one node at the same time. The information of the completed replication node is recorded to the initial replicated target data node, so that the progress of the entire replication task can be summarized at the initial replicated target data node.

[0102] S405, repeating the process of using the latest obtained replicated target data node to replicate to the node without target data until each node in the cluster has replicated the target data.

[0103] The method for spreading and copying large files in a cluster of the present invention aims to solve the problem of how to quickly copy target data to all nodes under the condition of limiting bandwidth. When there is no target data and after copying the target data, it can also be used as a copy source to copy to the node without target data, so that the copy task of the target data copying changes from one to many, and the copying process generates exponential fission through this operation, and the progress of file copying is significantly shortened. Therefore, the present invention implements a method for fast and controllable target data transmission in a cluster. It has the following advantages: First: From a macroscopic point of view, the nodes are divided into two parts with target data and without target data, so as to achieve exponential growth in the number of copies, make full use of the bandwidth resources of the nodes that have completed the copying, and the copying time is reduced from the original n to the current lg (n) 2. The entire program implementation language completely uses shell scripts, and does not strictly require the operating environment. Second: Use numbered cluster nodes for sorting and classification to achieve order and non-conflict in the subsequent copying process. Third: The number of single simultaneous copies at each node of the copied target data can be controlled by logic, which can reduce the impact of bandwidth usage in the copying process on the cluster.

[0104] The data replication device provided by the embodiment of the present invention is introduced below. The data replication device described below and the data replication method described above can be referred to each other.

[0105] Figure 5 A schematic diagram of the structure of a data replication device provided by an embodiment of the present invention may include:

[0106] The node determination module 100 is used to determine the initial replicated target data node; wherein the initial replicated target data node is a node that has target data, and the target data is the data that each node needs to replicate;

[0107] A first replication module 200 is used to replicate the target data to a preset number of first batch of target-free data nodes using the initial replicated target data nodes, to obtain a preset number of first batch of replicated target data nodes; wherein the preset number is a number determined according to node performance;

[0108] A second replication module 300 is used to use the initial replicated target data nodes and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of non-target data nodes, so as to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of non-target data nodes;

[0109] The third replication module 400 is used to repeatedly execute the process of using the replicated target data node to replicate the target data to the node without target data until all nodes have replicated the target data; wherein the replicated target data node is a node in the current cluster that has replicated the target data.

[0110] Further, based on the above embodiment, the above first copy module 200 may include:

[0111] A numbering unit is used to number the nodes according to the node address or the node host name to obtain the serial number corresponding to each node;

[0112] The first replication unit is used to replicate the target data to a preset number of the first batch of target-free data nodes using the initial replicated target data node according to the sequence number corresponding to each node, so as to obtain a preset number of the first batch of replicated target data nodes.

[0113] Further, based on the above embodiment, the preset number is x, the number of replication rounds is n, the base X=x+1, and the serial number of the replicated target data node is m; the above data replication device may further include:

[0114] The replication mapping relationship determination module is used to determine the number of nodes of the replicated target data nodes when the nth round of replication is , when the target data is currently being copied, the copy mapping relationship is determined as follows: the sequence number of the node without target data corresponding to the copied target data node with sequence number m is , Y is an integer from 1 to x;

[0115] The replication relationship determination module is used to use the replication mapping relationship to replicate the target data on each replicated target data node to the corresponding non-target data node.

[0116] Furthermore, the second copy module 300 may include:

[0117] The second replication unit is used to use an adjacent replication method to copy the target data to the second batch of non-target data nodes according to the initial replicated target data nodes and the first batch of replicated target data nodes, so as to obtain the second batch of replicated target data nodes; wherein, the adjacent replication method is that the non-target data nodes corresponding to each replicated target data node are a preset number of adjacent nodes.

[0118] Further, based on the above embodiment, the numbering unit may include:

[0119] The sequence number determination subunit is used to use the sequence number processing command to number the nodes through the pipeline symbol to obtain the sequence number corresponding to each node.

[0120] Further, based on any of the above embodiments, the above data replication device may further include:

[0121] A replication thread determination module is used to store all replication threads using a thread list; wherein the thread list includes replicated target data nodes, non-target data nodes and replication results; the replication result is the result of whether the target data has been replicated; the replication thread is the process of replicating the target data from the replicated target data node to the non-target data node;

[0122] A result dictionary determination module, used to record the replication result corresponding to the node in the thread list into a result dictionary; wherein the result dictionary uses the node name as a key and the replication result as a value;

[0123] A replication status determination module, used to traverse the result dictionary and determine the replication status corresponding to each replication thread according to the result dictionary; wherein the replication status includes replication success, replication in progress and replication error;

[0124] The replication status determination module may include:

[0125] a copy thread number determination unit, used to determine whether the number of successful copies in the result dictionary is equal to the number of current copy threads;

[0126] A copy completion determination unit, configured to determine that the copying is completed and execute the next round of copying when the number of successful copies in the result dictionary is equal to the number of current copying threads;

[0127] a replication error determination unit, configured to determine whether the replication status corresponding to the remaining threads is a replication error when the number of successful replications in the result dictionary is not equal to the number of current replication threads;

[0128] A replication error no-target data determining unit, configured to determine a no-target data node corresponding to the replication error when the replication status corresponding to the remaining threads is a replication error;

[0129] The replication is normally performed and the waiting replication unit is used to determine that the waiting replication is completed when the replication status corresponding to the remaining threads is not a replication error.

[0130] Further, based on any of the above embodiments, the above data replication device may further include:

[0131] A replication process information determination module, configured to obtain replication process information by using the information of the replication node stored in the initial replicated target data node;

[0132] A judgment module is used to determine whether to execute the next round of replication according to the replication process information; wherein the replication process information is information on whether the replication process is completed.

[0133] It should be noted that the order of the modules and units in the above data replication device can be changed without affecting the logic.

[0134] Figure 5 The description of the features in the corresponding embodiments can be found in Figure 5 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0135] The data replication device provided by the embodiment of the present invention may include: a node determination module 100, which is used to determine an initial replicated target data node; wherein the initial replicated target data node is a node with existing target data, and the target data is data that needs to be replicated by each node; a first replication module 200, which is used to use the initial replicated target data node to replicate the target data to a preset number of first batch of target-free nodes, to obtain a preset number of first batch of replicated target data nodes; wherein the preset number is a number determined according to node performance; a second replication module 300, which is used to use the initial replicated target data node and the first batch of replicated target data nodes to respectively replicate the target data to corresponding second batch of target-free nodes, to obtain a second batch of replicated target data nodes; wherein each node can replicate the target data to the preset number of target-free nodes; a third replication module 400, which is used to repeatedly execute the process of using the replicated target data node to replicate the target data to the target-free nodes until all nodes have replicated the target data; wherein the replicated target data node is a node in the current cluster that has replicated the target data. The embodiment of the present invention makes full use of the nodes that have replicated the target data in each round and uses them as replication sources to transmit files to other nodes that do not have the target data. Since the number of nodes that have replicated the target data is constantly increasing, exponentially growing replication is achieved, which shortens the replication time and improves the efficiency of file replication.

[0136] A data replication device provided by an embodiment of the present invention is introduced below. The data replication device described below and the data replication method described above can be referred to each other.

[0137] Figure 6 A schematic diagram of the structure of a data replication device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the data replication device includes: a memory 60 for storing a computer program;

[0138] The processor 61 is used to implement the steps of the data replication method in the above embodiment when executing a computer program.

[0139] The data replication device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0140] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0141] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, the relevant steps of the data replication method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to copying data, etc.

[0142] In some embodiments, the data replication device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0143] Those skilled in the art will understand that Figure 6The structure shown in the figure does not constitute a limitation on the data replication device, and may include more or less components than shown in the figure.

[0144] It is understandable that if the data replication method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0145] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data replication method are implemented.

[0146] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the data replication method when executed by a processor.

[0147] The above is a detailed introduction to a data replication method, device, equipment and computer-readable storage medium provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0148] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0149] The above is a detailed introduction to a data replication method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A data replication method, characterized in that: include: Determine an initial replicated target data node; wherein the initial replicated target data node is a node that has target data, and the target data is data that needs to be replicated for each node; Using the initial replicated target data node, the target data is replicated to a preset number of first batch nodes without target data, to obtain a preset number of first batch replicated target data nodes; wherein the preset number is a number determined according to node performance; Using the initial replicated target data nodes and the first batch of replicated target data nodes, the target data are respectively replicated to the corresponding second batch of non-target data nodes to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of non-target data nodes; Repeat the process of using replicated target data nodes to replicate the target data to nodes without target data until all nodes have replicated the target data; wherein the replicated target data nodes are nodes in the current cluster that have replicated the target data.

2. The data replication method according to claim 1, characterized in that: Using the initial replicated target data node to replicate the target data to a preset number of first batch nodes without target data, to obtain a preset number of first batch replicated target data nodes, including: Number the nodes according to the node address or node host name to obtain the serial number corresponding to each node; The target data is copied to a preset number of the first batch of target-free data nodes using the initial copied target data nodes according to the sequence number corresponding to each node, so as to obtain a preset number of the first batch of copied target data nodes.

3. The data replication method according to claim 2, characterized in that: The preset number is x, the number of replication rounds is n, the base number X=x+1, and the serial number of the replicated target data node is m; When the nth round of replication is completed, the number of nodes that have replicated the target data node is , when the target data is currently being copied, the copy mapping relationship is determined as follows: the sequence number of the node without target data corresponding to the copied target data node with sequence number m is , Y is an integer from 1 to x; The target data on each replicated target data node is replicated to a corresponding non-target data node using the replication mapping relationship.

4. The data replication method according to claim 2, characterized in that: The method of using the initial replicated target data nodes and the first batch of replicated target data nodes to respectively replicate the target data to corresponding second batch of target-free data nodes to obtain a second batch of replicated target data nodes includes: The target data is copied to the second batch of non-target data nodes respectively according to the initial replicated target data nodes and the first batch of replicated target data nodes by using an adjacent replication method to obtain the second batch of replicated target data nodes; wherein, the adjacent replication method is that the non-target data nodes corresponding to each replicated target data node are a preset number of adjacent nodes.

5. The data replication method according to claim 2, characterized in that: The node numbering according to the node address or the node host name to obtain the sequence number corresponding to each node includes: The nodes are numbered by using a sequence number processing command through a pipe symbol to obtain the sequence number corresponding to each node.

6. The data replication method according to any one of claims 1 to 5, characterized in that: The copying process also includes: All replication threads are stored using a thread list; wherein the thread list includes replicated target data nodes, non-target data nodes, and replication results; the replication result is the result of whether the target data has been replicated; the replication thread is the process of replicating the target data from the replicated target data node to the non-target data node; Record the copy result corresponding to the node in the thread list into a result dictionary; wherein the result dictionary uses the node name as a key and the copy result as a value; Traversing the result dictionary, and determining the replication status corresponding to each replication thread according to the result dictionary; wherein the replication status includes replication success, replication in progress, and replication error; Wherein, determining the replication status according to the result dictionary includes: Determine whether the number of successful copies in the result dictionary is equal to the number of current copy threads; When the number of successful copies in the result dictionary is equal to the number of current copy threads, it is determined that the copying is complete and the next round of copying is performed; When the number of successful copies in the result dictionary is not equal to the number of current copy threads, determining whether the copy status corresponding to the remaining threads is a copy error; When the replication status corresponding to the remaining threads is a replication error, determining a target data node corresponding to the replication error; When the replication status corresponding to the remaining threads is not replication error, it is determined to wait for the replication to end.

7. The data replication method according to any one of claims 1 to 5, characterized in that: The copying process also includes: Using the initial replicated target data node to store the completed replication node information, obtaining replication process information; Determine whether to execute the next round of replication according to the replication progress information; wherein the replication progress information is information on whether the replication process is completed.

8. A data replication device, characterized in that: include: A node determination module is used to determine an initial replicated target data node; wherein the initial replicated target data node is a node that has target data, and the target data is data that needs to be replicated for each node; A first replication module is used to replicate the target data to a preset number of first batch nodes without target data by using the initial replicated target data node, so as to obtain a preset number of first batch replicated target data nodes; wherein the preset number is a number determined according to node performance; A second replication module is used to use the initial replicated target data node and the first batch of replicated target data nodes to respectively replicate the target data to the corresponding second batch of non-target data nodes to obtain a second batch of replicated target data nodes; wherein each node is capable of replicating the target data to the preset number of non-target data nodes; The third replication module is used to repeatedly execute the process of using the replicated target data node to replicate the target data to the node without target data until all nodes have replicated the target data; wherein the replicated target data node is a node in the current cluster that has replicated the target data.

9. A data replication device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data replication method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data replication method according to any one of claims 1 to 7 are implemented.