A distributed file synchronization method, system, device and storage medium

Abstract

This application discloses a distributed file synchronization method, system, device, and storage medium. The technical solution provided by this application obtains the node startup time of each regional node in the distributed system, determines the node identity information of each regional node based on the node startup time, and performs file synchronization of the distributed system based on the node identity information, thereby achieving decentralized file synchronization. Even with changes in the network environment, efficient file synchronization can be achieved through changes in node identity information, simplifying the cumbersome process of file synchronization between nodes in the distributed system and improving the stability and reliability of file synchronization.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a distributed file synchronization method, system, device, and storage medium. Background Technology

[0002] Currently, in distributed system operation scenarios, file synchronization between nodes is frequently required. File synchronization technology is a technique that ensures the consistency of file content among multiple nodes by comparing and transferring file data. In related file synchronization solutions, one node typically proactively notifies other nodes to pull files from the data source to achieve file synchronization between nodes.

[0003] However, related file synchronization schemes require each node to frequently pull files from the data source, leading to significant network resource consumption, network congestion, and latency. The entire process is cumbersome and time-consuming, impacting file synchronization efficiency. Furthermore, because file synchronization is performed in a centralized manner, a single point of failure in the central node responsible for notifying other nodes can cause file synchronization to fail, affecting its stability and reliability. Summary of the Invention

[0004] This application provides a distributed file synchronization method, system, device, and storage medium that can adapt to changes in the network environment to achieve efficient file synchronization between nodes, solving the technical problem of cumbersome and time-consuming file synchronization process between nodes in a distributed system.

[0005] In a first aspect, embodiments of this application provide a distributed file synchronization method, comprising:

[0006] Obtain the node startup time of each regional node in the distributed system, and determine the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node.

[0007] When the node identity information on the local machine is the primary leader node, the update file of the data source is received and the update file is stored in the local first file information. The first difference file information between the first file information and the second file information of each regional leader node is determined, and the first difference file information is synchronized to the corresponding regional leader node.

[0008] If the node identity information of the local machine is a regional leader node, determine the second difference file information between the local second file information and the third file information of each regional follower node in the target region, and synchronize the second difference file information to the regional follower nodes.

[0009] When the node identity information of the local machine is a region follower node, it reports the local third file information to the regional leader node of the target region and receives the second difference file information synchronized by the regional leader node.

[0010] In a second aspect, embodiments of this application provide a distributed file synchronization system, including:

[0011] The identity determination module is configured to obtain the node startup time of each regional node in the distributed system, and determine the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node.

[0012] The first synchronization module is configured to receive the update file from the data source and store the update file in the first file information on the local machine when the node identity information of the local machine is the main leader node, determine the first difference file information between the first file information and the second file information of each regional leader node, and synchronize the first difference file information to the corresponding regional leader node.

[0013] The second synchronization module is configured to, when the node identity information of the local machine is a regional leader node, determine the second difference file information between the local second file information and the third file information of each regional follower node in the target region, and synchronize the second difference file information to the regional follower node.

[0014] The third synchronization module is configured to report local third file information to the regional leader node of the target region when the node identity information of the local machine is a regional follower node, and receive the second difference file information synchronized by the regional leader node.

[0015] In a third aspect, embodiments of this application provide a distributed file synchronization device, comprising:

[0016] Memory and one or more processors;

[0017] The memory is configured to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed file synchronization method as described in the first aspect.

[0019] In a fourth aspect, embodiments of this application provide a non-volatile computer-readable storage medium storing computer-executable instructions configured, when executed by a computer processor, to perform the distributed file synchronization method as described in the first aspect.

[0020] In a fifth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the distributed file synchronization method as described in the first aspect.

[0021] This application embodiment obtains the node startup time of each regional node in the distributed system, and determines the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node. When the node identity information of the local machine is the master leader node, it receives the update file from the data source and stores the update file in the local first file information, determines the first difference file information between the first file information and the second file information of each regional leader node, and synchronizes the first difference file information to the corresponding regional leader node. When the node identity information of the local machine is the regional leader node, it determines the second difference file information between the local second file information and the third file information of each regional follower node in its target region, and synchronizes the second difference file information to the regional follower nodes. When the node identity information of the local machine is the regional follower node, it reports the local third file information to the regional leader node of its target region and receives the second difference file information synchronized by the regional leader node. By employing the aforementioned technical means, the node identity information of each regional node is determined through the node startup time. Based on the node identity information, file synchronization of the distributed system is performed, thereby achieving decentralized file synchronization. Even when the network environment changes, efficient file synchronization can still be achieved by changing the node identity information, simplifying the cumbersome process of file synchronization between distributed system nodes and improving the stability and reliability of file synchronization. Attached Figure Description

[0022] Figure 1 This is a flowchart of a distributed file synchronization method provided in an embodiment of this application;

[0023] Figure 2This is an interaction flowchart of each module of the regional node in the embodiments of this application;

[0024] Figure 3 This is a flowchart illustrating the file synchronization process of various regional nodes in the distributed system as described in this application embodiment;

[0025] Figure 4 This is a schematic diagram of the structure of a distributed file synchronization system provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a distributed file synchronization device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] The distributed file synchronization method provided in this application aims to determine the node identity information of each regional node by means of the node startup time, and to perform file synchronization of the distributed system based on the node identity information, thereby achieving decentralized file synchronization.

[0029] In related file synchronization schemes, one node typically proactively notifies other nodes to pull files from the data source. This approach has significant drawbacks: frequent file pull operations consume substantial network resources from the data source, especially when there are many nodes or data updates are frequent, potentially leading to network congestion and latency. Alternatively, incremental transmission methods can reduce network resource consumption, but the transmitting node needs to actively obtain information from other nodes requiring synchronization, resulting in a lack of flexibility and difficulty in handling real-time network topology changes.

[0030] Furthermore, related file synchronization solutions are typically centralized, requiring complex configuration and maintenance, and suffer from a series of inherent problems, such as single points of failure, limited system scalability, and inability to effectively handle network partitions. These shortcomings are particularly pronounced in large-scale or dynamic network environments. Although some distributed file synchronization solutions have emerged that can overcome the shortcomings of centralized systems to some extent, these solutions often rely on a distributed component, introducing additional management complexity and operational costs.

[0031] Based on this, a distributed file synchronization method according to an embodiment of this application is provided to solve the technical problem of the cumbersome and time-consuming file synchronization process between nodes in a distributed system.

[0032] Example:

[0033] Figure 1 A flowchart of a distributed file synchronization method provided in an embodiment of this application is given. The distributed file synchronization method provided in this embodiment can be executed by a distributed file synchronization device, which can be implemented by software and / or hardware. The distributed file synchronization device can consist of two or more physical entities, or it can consist of a single physical entity. Generally, the distributed file synchronization device can be a node server or other processing device in a distributed system.

[0034] The following description uses the distributed file synchronization device as an example to illustrate the distributed file synchronization method. (Refer to...) Figure 1 The distributed file synchronization method specifically includes:

[0035] S110. Obtain the node startup time of each regional node in the distributed system, and determine the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node.

[0036] This application achieves seamless device discovery and efficient file synchronization within a local area network through a decentralized design when performing file synchronization in a distributed system. Simultaneously, the decentralized approach allows for real-time adaptation to changes in the network environment, supports seamless collaboration among multiple nodes, reduces network resource consumption, and thus improves overall performance and stability.

[0037] In the process of file synchronization between nodes in a distributed system, the node startup time of each node is obtained to determine its identity. This node identity information is then used for file synchronization. Each node in the distributed system records its startup time. By obtaining these startup times, a corresponding role can be assigned to each node. The node with the earliest startup time is selected as the master leader node. Within each target region, the node with the earliest startup time becomes the region leader node, and the remaining nodes act as region follower nodes for their respective target regions. The distributed system can be divided into multiple regions according to actual needs, with each region configured with a corresponding number of region nodes. The defined region is the target region.

[0038] Each region node includes the following modules:

[0039] Node discovery module: Used to discover other nodes within the local area network in real time based on mDNS;

[0040] Node management module: Used to maintain the file synchronization status and health status of other nodes, responsible for electing a leader node, and providing relevant query interfaces to users;

[0041] File download module: Used to download files from a data source;

[0042] File management module: Used to maintain the file status of this node;

[0043] Transmission optimization module: Used to support incremental synchronization and breakpoint resumption, ensuring that bandwidth is saved as much as possible and efficiency is improved during file transfer;

[0044] Node state machine: It is responsible for the logic related to the switching between different states of this node.

[0045] like Figure 2 As shown, after a regional node starts up, its node discovery module sends mDNS advertisement messages to the multicast address of the specified IP and port to register a specific hostname and the node's IP address. mDNS (Multicast DNS) is a device discovery protocol within a local area network (LAN) that achieves automatic device discovery by broadcasting DNS requests and responses within the LAN. Subsequently, the node discovery module periodically sends mDNS query messages to query other devices with specific hostnames to obtain information about other nodes within the current LAN. When an mDNS query response is received, the retrieved node information is handed over to the node management module for processing.

[0046] The node management module periodically requests information from nodes in other regions, such as their system startup time, region information, and health status. Based on the obtained startup times of each regional node, the module can configure the node identity information of each regional node according to its startup time and elect a corresponding leader node for file synchronization in the distributed system.

[0047] By using the mDNS protocol, distributed systems can reduce system complexity, simplify the connection process between nodes, and reduce reliance on centralized management, thereby improving the flexibility and reliability of file synchronization, eliminating the need for distributed systems to actively obtain node information and introduce additional components such as a registry center.

[0048] Specifically, the node identity information in this application includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the primary leader node corresponding to each regional leader node. When determining the node identity information of each regional node based on its startup time, the following is included:

[0049] A min-heap data structure is constructed based on the node startup time of each region node;

[0050] The system queries the min-heap data structure based on a pre-built comparator and determines the node identity information of each region node according to its position in the min-heap data structure.

[0051] The election of the leader node is achieved by maintaining a min-heap sorted by node startup time. A comparator is defined to compare node startup times, prioritizing nodes with shorter startup times. If startup times are the same, node IPs are compared, with shorter IPs taking precedence. Furthermore, an election class is defined based on a priority queue (in C++, this can inherit from `std::priority_queue` from the standard library), using the aforementioned comparator as the sorting rule.

[0052] During leader node election, the root node is selected from the priority queue of the min-heap to become the regional leader node for the corresponding target region, and the head of the min-heap becomes the master leader node for each regional leader node. The remaining nodes become regional follower nodes under the corresponding regional leader node (root node). Based on the election results, node identity information is assigned to each node, including regional follower node, regional leader node, and master leader node.

[0053] It is understood that the regional leader node is the node with the earliest system startup time among the nodes in its region, and the primary leader node is the node with the earliest system startup time among all nodes. When a node goes offline, a new leader node election will be held. When another node in the target region becomes the leader node, the other regional nodes in this region will become regional follower nodes.

[0054] After constructing a min-heap data structure based on the node startup time of each region node, the following is also included:

[0055] If a new region node is detected, insert the new region node at the tail of the min-heap data structure based on the node's startup time, and rebuild the min-heap data structure.

[0056] If the removal of a specified region node is detected, and the specified region node is at the head of the min-heap data structure, the specified region node is removed based on the node pop function. If the specified region node is not at the head of the min-heap data structure, the specified region node is removed directly, and the min-heap data structure is reconstructed.

[0057] The node discovery module queries the node information of other regional nodes within the distributed system in real time. When a new regional node is detected, it inserts the new node into the tail of the priority queue and rebuilds a min-heap. Furthermore, when a regional node is detected as being removed from the distributed system, it searches the priority queue for the node corresponding to that regional node's IP address. If the regional node is found at the head of the queue, it is popped from the queue. Otherwise, it is removed from the queue, and the min-heap is rebuilt.

[0058] Regional nodes maintain their state transitions through a node state machine. Nodes have three states: initial state, follower state, and leader state. A node is in the initial state when it hasn't yet obtained information from other nodes. Once a node obtains information from other nodes, it can elect a local leader to determine whether it is a leader or a follower, thus establishing its node identity.

[0059] In addition, the node management module is used to view the IP addresses and current connection status of regional nodes, check the progress and history of file synchronization, detect the health status of other nodes (such as online status and last synchronization time), pause or resume synchronization tasks for specific nodes, and avoid file conflicts caused by network instability. When the health status of a node becomes abnormal, other nodes can promptly detect the abnormal node through mDNS broadcast and suspend synchronization operations to the abnormal node. When a new node is discovered, the node management module will also immediately obtain the relevant information of the new node and add it to the management list. The distributed system can proactively monitor the health status, synchronization progress, and recent file synchronization records of other nodes in the network without additional configuration, making file synchronization more convenient, efficient, and highly stable.

[0060] S120. When the node identity information of the local machine is the primary leader node, receive the update file from the data source and store the update file in the local first file information, determine the first difference file information between the first file information and the second file information of each regional leader node, and synchronize the first difference file information to the corresponding regional leader node.

[0061] Furthermore, based on the aforementioned node identity information configuration, if the current regional node's local node identity information is that of a primary leader node, then when the data source has updated files, the primary leader node receives these files and updates its first file information database. Subsequently, the primary leader node calculates the difference between the first file information and the second file information of each regional leader node, defining this difference as the first difference file information. The primary leader node then sends this difference file information to the corresponding regional leader nodes to synchronize files between the primary leader node and the regional leader nodes.

[0062] S130. If the node identity information of the local machine is a regional leader node, determine the second difference file information between the local second file information and the third file information of each regional follower node in the target region, and synchronize the second difference file information to the regional follower node.

[0063] Based on the node identity information configuration described above, if the current regional node's local node identity information is that of a regional leader node, the regional leader node will receive the first difference file information from the primary leader node and update its own second file information database. Next, the regional leader node calculates the difference between the second file information and the third file information of each regional follower node within its region; this is the second difference file information. The regional leader node then sends this difference file information to its respective target regional follower nodes to perform file synchronization between the regional leader node and the regional follower nodes.

[0064] S140. If the node identity information of the local machine is a region follower node, report the local third file information to the region leader node of the target region, and receive the second difference file information synchronized by the region leader node.

[0065] Based on the above node identity information configuration, if the node identity information of the current regional node is a regional follower node, the regional follower node will periodically or as needed report its third file information to the regional leader node of its region. When it receives the second difference file information from the regional leader node, the regional follower node updates its own third file information database to synchronize files between the regional leader node and the regional follower node.

[0066] For example, refer to Figure 3In a distributed file system, after determining node identity based on node startup time, node 1 is configured as the primary leader node and also serves as the regional leader node for its assigned region, communicating with the regional follower nodes Nodes 2 and 3. Nodes 4 and 5 are regional leader nodes. During file synchronization, node 1 downloads updated files from the data source and compares them with the local file information of each regional leader node to determine which files need updating and update them on nodes 4 and 5. Then, as the regional leader node, node 1 synchronizes its files by comparing the differences between its own files and those of nodes 2 and 3. Node 4 synchronizes its files by comparing the differences between its own files and those of nodes 5 and 6. Node 7 synchronizes its files by comparing the differences between its own files and those of nodes 8 and 9.

[0067] In a distributed system, nodes significantly reduce data transmission volume, network resource consumption, network congestion, and latency by transmitting only differentiated file information instead of the entire file. Simultaneously, differentiated synchronization strategies reduce unnecessary data transmission and processing, accelerating file synchronization and enabling nodes in the distributed system to achieve file content consistency more quickly. Furthermore, the distributed leader node architecture reduces the risk of single points of failure. Even if a leader node fails, the system can quickly recover and continue operating, ensuring the stability and reliability of file synchronization. In addition, decentralized node identity configuration can dynamically adapt to changes in network conditions, such as bandwidth fluctuations and latency variations, optimizing performance by adjusting synchronization frequency and strategies. Thus, through intelligent node role allocation and differentiated file synchronization strategies, the efficiency, stability, and reliability of file synchronization in a distributed system are significantly improved.

[0068] Optionally, the first file information, the second file information, and the third file information all include the corresponding file type and file update time;

[0069] The first difference file information between the first file information and the second file information of each regional leadership node is determined, including:

[0070] The first file increment between the first file information and the second file information of each regional leader node is determined based on the file type and file update time, and the first file increment is used as the first difference file information between the first file information and the second file information of each regional leader node.

[0071] Determine the second difference file information between the local second file information and the third file information of each region following the target region, including:

[0072] The second file increment is determined based on the file type and file update time, relative to the third file information of each region's following nodes. This second file increment is used as the second difference file information between the second file information and the third file information of each region's following nodes in the target region.

[0073] The file management module periodically retrieves the parent node's information from the node discovery module and then sends the file information for this node, including file type, file update time, and file MD5 hash. Upon receiving the file information from the follower nodes, the parent node compares it with its own file information and then synchronizes any discrepancies to the next level node.

[0074] For the primary leader node, file types and update times are compared to determine which files exist in the first file information but not in the second file information of the regional leader nodes, or which files exist but have different content (determined by update time). These different or newly added file information constitute the first file increment. The first file increment is considered the first difference between the first file information and the second file information of each regional leader node.

[0075] For a regional leader node, the differences between its local second file information and the third file information of each regional follower node in the target region are determined. By comparing file types and file update times, file information that exists in the local second file information but not in the third file information of the regional follower nodes, or that exists in both but has different content, is identified, forming the second file increment. The second file increment is considered as the second difference file information between the second file information and the third file information of each regional follower node in the target region.

[0076] After receiving the second difference file information synchronized from the regional leader node, the following is also included:

[0077] MD5 verification is performed based on the second difference file information and the local third file information, and the third file information is then updated synchronously based on the MD5 verification result.

[0078] After receiving synchronized files from the regional leader node, the regional follower node verifies the MD5 hash of the files before writing them to disk. The file management module also periodically checks the MD5 hash of local files to prevent tampering. If an incorrect MD5 hash is found, the regional leader node is required to resynchronize. Similarly, the regional leader node performs the same MD5 verification process when synchronizing files from the primary leader node. Furthermore, after the primary leader node finishes downloading updated files, it also verifies the MD5 hash of the files before handing them over to the file management module for disk writing.

[0079] Optionally, synchronizing the first difference file information to the corresponding regional leader node also includes:

[0080] Detect the health status of the corresponding regional leader node, and pause or resume the synchronization task of the first difference file information based on the node health status;

[0081] Synchronizing the second difference file information to the region following node also includes:

[0082] Detect the health status of the corresponding region's following nodes, and pause or resume the synchronization task of the second difference file information based on the node's health status.

[0083] Before synchronizing the first difference file information to the corresponding regional leader nodes, the health status of these regional leader nodes is first checked. Node health status can include various information such as node online status, resource usage (e.g., CPU, memory, disk space), and network connectivity. Based on the detected node health status, the primary leader node makes a decision: whether to continue, pause, or resume the synchronization task of the first difference file information. If a node is detected to be in an unhealthy state (e.g., excessive resource usage, network instability, or node crash), the primary leader node can pause the synchronization task to avoid placing a greater burden on the node or causing synchronization failure. When the regional leader nodes recover their health status, the primary leader node automatically resumes the synchronization task.

[0084] Similarly, before synchronizing the second difference file information to the corresponding regional follower nodes, the regional leader node also checks the health status of these regional follower nodes. Based on the detected node health status, the regional leader node decides whether to continue, pause, or resume the synchronization task of the second difference file information. If a regional follower node is in an unhealthy state, the regional leader node will pause the synchronization task. Once the regional follower node recovers its health, the synchronization task can resume.

[0085] By adding node health status detection and synchronization task management, the synchronization tasks of different file information can be handled more intelligently, thereby improving the success rate of file synchronization and avoiding unnecessary burden on unhealthy nodes.

[0086] As described above, by obtaining the node startup time of each regional node in the distributed system, the node identity information of each regional node is determined based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes within its target region, and the master leader node is used to communicate with each regional leader node. When the node identity information of the local machine is the master leader node, the update file of the data source is received and the update file is stored in the local first file information. The first difference file information between the first file information and the second file information of each regional leader node is determined, and the first difference file information is synchronized to the corresponding regional leader node. When the node identity information of the local machine is the regional leader node, the second difference file information between the local second file information and the third file information of each regional follower node in the target region is determined, and the second difference file information is synchronized to the regional follower nodes. When the node identity information of the local machine is the regional follower node, the local third file information is reported to the regional leader node of the target region, and the second difference file information synchronized by the regional leader node is received. By employing the aforementioned technical means, the node identity information of each regional node is determined through the node startup time. Based on the node identity information, file synchronization of the distributed system is performed, thereby achieving decentralized file synchronization. Even when the network environment changes, efficient file synchronization can still be achieved by changing the node identity information, simplifying the cumbersome process of file synchronization between distributed system nodes and improving the stability and reliability of file synchronization.

[0087] Based on the above embodiments, Figure 4 This is a schematic diagram of the structure of a distributed file synchronization system provided in this application. (Reference) Figure 4 The distributed file synchronization system provided in this embodiment specifically includes: an identity determination module 21, a first synchronization module 22, a second synchronization module 23, and a third synchronization module 24.

[0088] Among them, the identity determination module 21 is configured to obtain the node startup time of each regional node in the distributed system, and determine the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node.

[0089] The first synchronization module 22 is configured to receive the update file from the data source and store the update file in the first file information on the local machine when the node identity information of the local machine is the main leader node, determine the first difference file information between the first file information and the second file information of each regional leader node, and synchronize the first difference file information to the corresponding regional leader node.

[0090] The second synchronization module 23 is configured to, when the node identity information of the local machine is a regional leader node, determine the second difference file information between the local second file information and the third file information of each regional follower node in the target region, and synchronize the second difference file information to the regional follower node.

[0091] The third synchronization module 24 is configured to report local third file information to the regional leader node of the target region when the node identity information of the local machine is a regional follower node, and receive the second difference file information synchronized by the regional leader node.

[0092] Specifically, the node identity information of each regional node is determined based on the node startup time, including:

[0093] A min-heap data structure is constructed based on the node startup time of each region node;

[0094] The system queries the min-heap data structure based on a pre-built comparator and determines the node identity information of each region node according to its position in the min-heap data structure.

[0095] After constructing a min-heap data structure based on the node startup time of each region node, the following is also included:

[0096] If a new region node is detected, insert the new region node at the tail of the min-heap data structure based on the node's startup time, and rebuild the min-heap data structure.

[0097] If the removal of a specified region node is detected, and the specified region node is at the head of the min-heap data structure, the specified region node is removed based on the node pop function. If the specified region node is not at the head of the min-heap data structure, the specified region node is removed directly, and the min-heap data structure is reconstructed.

[0098] Specifically, the first file information, the second file information, and the third file information all include the corresponding file type and file update time;

[0099] The first difference file information between the first file information and the second file information of each regional leadership node is determined, including:

[0100] The first file increment between the first file information and the second file information of each regional leader node is determined based on the file type and file update time, and the first file increment is used as the first difference file information between the first file information and the second file information of each regional leader node.

[0101] Determine the second difference file information between the local second file information and the third file information of each region following the target region, including:

[0102] The second file increment is determined based on the file type and file update time, relative to the third file information of each region's following nodes. This second file increment is used as the second difference file information between the second file information and the third file information of each region's following nodes in the target region.

[0103] Specifically, synchronizing the first difference file information to the corresponding regional leader node also includes:

[0104] Detect the health status of the corresponding regional leader node, and pause or resume the synchronization task of the first difference file information based on the node health status;

[0105] Synchronizing the second difference file information to the region following node also includes:

[0106] Detect the health status of the corresponding region's following nodes, and pause or resume the synchronization task of the second difference file information based on the node's health status.

[0107] Specifically, after receiving the second difference file information synchronized from the regional leader node, the process also includes:

[0108] MD5 verification is performed based on the second difference file information and the local third file information, and the third file information is then updated synchronously based on the MD5 verification result.

[0109] As described above, by obtaining the node startup time of each regional node in the distributed system, the node identity information of each regional node is determined based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes within its target region, and the master leader node is used to communicate with each regional leader node. When the node identity information of the local machine is the master leader node, the update file of the data source is received and the update file is stored in the local first file information. The first difference file information between the first file information and the second file information of each regional leader node is determined, and the first difference file information is synchronized to the corresponding regional leader node. When the node identity information of the local machine is the regional leader node, the second difference file information between the local second file information and the third file information of each regional follower node in the target region is determined, and the second difference file information is synchronized to the regional follower nodes. When the node identity information of the local machine is the regional follower node, the local third file information is reported to the regional leader node of the target region, and the second difference file information synchronized by the regional leader node is received. By employing the aforementioned technical means, the node identity information of each regional node is determined through the node startup time. Based on the node identity information, file synchronization of the distributed system is performed, thereby achieving decentralized file synchronization. Even when the network environment changes, efficient file synchronization can still be achieved by changing the node identity information, simplifying the cumbersome process of file synchronization between distributed system nodes and improving the stability and reliability of file synchronization.

[0110] The distributed file synchronization system provided in this application embodiment can be configured to execute the distributed file synchronization method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0111] Based on the above practical examples, this application also provides a distributed file synchronization device, referring to... Figure 5The distributed file synchronization device includes a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the distributed file synchronization method described in any embodiment of this application (e.g., an identity verification module, a first synchronization module, a second synchronization module, and a third synchronization module in a distributed file synchronization system). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory, thereby implementing the aforementioned distributed file synchronization method. The input device can be configured to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device. The output device may include a display screen or other display device. The distributed file synchronization device provided above can be configured to execute the distributed file synchronization method provided in the above embodiments, possessing corresponding functions and beneficial effects.

[0112] Based on the above embodiments, this application also provides a non-volatile computer-readable storage medium storing computer-executable instructions. These computer-executable instructions, when executed by a computer processor, are configured to perform a distributed file synchronization method. The storage medium can be any type of memory device or storage device. Of course, the computer-executable instructions of the non-volatile computer-readable storage medium provided in this application are not limited to the distributed file synchronization method described above; they can also perform related operations in the distributed file synchronization method provided in any embodiment of this application.

[0113] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the distributed file synchronization method described in the various embodiments of this application.

Claims

1. A distributed file synchronization method, characterized in that, include: The node startup time of each regional node in the distributed system is obtained, and the node identity information of each regional node is determined based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each regional leader node. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each regional leader node. When the node identity information of the local machine is the main leader node, the update file of the data source is received and the update file is stored in the first file information of the local machine. The first difference file information between the first file information and the second file information of each of the regional leader nodes is determined, and the first difference file information is synchronized to the corresponding regional leader node. If the node identity information of the local machine is the regional leader node, determine the second difference file information between the second file information of the local machine and the third file information of each of the regional follower nodes in the target region, and synchronize the second difference file information to the regional follower nodes; When the node identity information of the local machine is the region follower node, the local third file information is reported to the region leader node of the target region, and the second difference file information synchronized by the region leader node is received.

2. The distributed file synchronization method according to claim 1, characterized in that, The process of determining the node identity information of each regional node based on the node startup time includes: A min-heap data structure is constructed based on the node startup time of each region node; The min-heap data structure is queried based on a pre-built comparator, and the node identity information of each region node is determined according to the position of each region node in the min-heap data structure.

3. The distributed file synchronization method according to claim 2, characterized in that, After constructing the min-heap data structure based on the node startup time of each region node, the method further includes: If a new region node is detected, the new region node is inserted at the tail of the min-heap data structure based on the node's startup time, and the min-heap data structure is reconstructed. If a node in a specified region is detected to be removed, and the node is at the head of the min-heap data structure, the node is removed based on the node pop function. If the node is not at the head of the min-heap data structure, the node is removed directly, and the min-heap data structure is reconstructed.

4. The distributed file synchronization method according to claim 1, characterized in that, The first file information, the second file information, and the third file information all include the corresponding file type and file update time; The step of determining the first difference file information between the first file information and the second file information of each of the regional leadership nodes includes: Based on the file type and the file update time, a first file increment is determined between the first file information and the second file information of each of the regional leader nodes, and the first file increment is used as the first difference file information between the first file information and the second file information of each of the regional leader nodes. The determination of the second difference file information between the local second file information and the third file information of each of the region following nodes in the target region includes: The second file increment is determined based on the file type and the file update time, relative to the third file information of each of the regional following nodes. The second file increment is used as the second difference file information between the second file information and the third file information of each of the regional following nodes in the target region.

5. The distributed file synchronization method according to any one of claims 1-4, characterized in that, The step of synchronizing the first difference file information to the corresponding regional leader node further includes: Detect the node health status of the corresponding regional leader node, and pause or resume the synchronization task of the first difference file information according to the node health status; The step of synchronizing the second difference file information to the region following node further includes: Detect the node health status of the corresponding region following node, and pause or resume the synchronization task of the second difference file information according to the node health status.

6. The distributed file synchronization method according to any one of claims 1-4, characterized in that, After receiving the second difference file information synchronized by the regional leader node, the method further includes: Based on the second difference file information and the local third file information, perform MD5 verification, and based on the MD5 verification result, perform synchronous update processing of the third file information.

7. A distributed file synchronization system, characterized in that, include: The identity determination module is configured to obtain the node startup time of each regional node in the distributed system, and determine the node identity information of each regional node based on the node startup time. The node identity information includes regional follower nodes and regional leader nodes corresponding to different target regions, as well as the master leader node corresponding to each of the regional leader nodes. The regional leader node is used to communicate with other regional follower nodes in its target region, and the master leader node is used to communicate with each of the regional leader nodes. The first synchronization module is configured to, when the node identity information of the local machine is the main leader node, receive the update file of the data source and store the update file in the first file information of the local machine, determine the first difference file information between the first file information and the second file information of each of the regional leader nodes, and synchronize the first difference file information to the corresponding regional leader node. The second synchronization module is configured to, when the node identity information of the local machine is the regional leader node, determine the second difference file information between the second file information of the local machine and the third file information of each of the regional follower nodes in the target region, and synchronize the second difference file information to the regional follower nodes. The third synchronization module is configured to, when the node identity information of the local machine is the region following node, report the local third file information to the region leader node of the target region, and receive the second difference file information synchronized by the region leader node.

8. A distributed file synchronization device, characterized in that, include: Memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed file synchronization method as described in any one of claims 1-6.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by a computer processor, are configured to perform the distributed file synchronization method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the distributed file synchronization method as described in any one of claims 1-6.

Images