A distributed data synchronization method, device and storage medium

Abstract

This invention discloses a distributed data synchronization method, device, and storage medium, belonging to the field of data processing. It collects multi-dimensional state features of all nodes within a distributed cluster within a sliding time window, constructs a node feature matrix, calculates the comprehensive dynamic weight of each node based on the node feature matrix, and constructs a three-layer cluster architecture of global master node – regional master node – regional slave node. Based on the transaction log of the global master node, it extracts the operation set of incremental transactions, divides the incremental data into fixed-size blocks, and compresses the incremental data using strong and weak double hash matching and sliding window differential encoding to obtain compressed incremental synchronization data packets. Finally, it constructs a synchronization cost model between nodes, generates a minimum synchronization cost spanning tree with the global master node as the root node, obtains the optimal synchronization path, and completes the incremental data synchronization of the distributed cluster.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically to a distributed data synchronization method, device, and storage medium. Background Technology

[0002] With the rapid development of big data and cloud computing technologies, distributed systems have become the core architecture for enterprise-level data storage and processing. Distributed data synchronization is a core technology for ensuring data consistency, availability, and partition tolerance in distributed systems. Currently, mainstream distributed data synchronization technologies are mainly based on consensus algorithms such as Raft and Paxos, combined with Change Data Capture (CDC) to complete incremental data synchronization. However, in practical engineering applications, the following technical pain points remain unresolved: The consensus architecture suffers from significant performance bottlenecks: Traditional Raft / Paxos algorithms employ a fixed single master node architecture, where all write operations must be processed by the master node. The computing power and network bandwidth of the master node become the bottleneck for the synchronization performance of the entire cluster. Especially in cross-regional distributed clusters, the network latency between the master node and remote nodes is extremely high, resulting in low cluster synchronization throughput and high latency. The consensus process is limited by weak nodes, further reducing synchronization efficiency.

[0003] High bandwidth consumption for incremental data transmission: Most existing incremental synchronization solutions directly transmit the original changed data without efficiently compressing the incremental data based on its characteristics. In cross-regional synchronization scenarios, the bandwidth cost is extremely high. Some solutions use differential compression algorithms that are only based on full snapshots, which cannot adapt to high-frequency incremental update scenarios, resulting in low compression ratios and poor block matching efficiency. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a distributed data synchronization method, device, and storage medium that achieves quantitative optimization of the entire synchronization process, solving problems such as synchronization performance bottlenecks, high bandwidth consumption, and weak conflict resolution capabilities in existing technologies.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A distributed data synchronization method is provided, characterized by comprising the following steps: S1: Collect multi-dimensional state features of all nodes in the distributed cluster within the sliding time window, and normalize and positiveize all state features to construct a node feature matrix. S2: Calculate the comprehensive dynamic weight of each node based on the node feature matrix, and use the comprehensive dynamic weight to elect the global master node and regional deputy master node. Combine the network topology to divide the distributed cluster into regions and construct a three-layer cluster architecture of global master node - regional master node - regional deputy node. S3: Based on the transaction log of the global master node, change data is captured and the operation set of incremental transactions is extracted; the incremental data is divided into fixed-size blocks, and the incremental data is compressed by strong and weak double hash matching and sliding window differential encoding to obtain compressed incremental synchronization data packets. S4: Based on the three-layer cluster architecture and the comprehensive dynamic weight of nodes, a synchronization cost model between nodes is constructed. The minimum synchronization cost spanning tree is generated with the global master node as the root node to obtain the optimal synchronization path. Incremental synchronization data packets are transmitted to the regional master node and regional slave node using the optimal synchronization path to complete the incremental data synchronization of the distributed cluster.

[0006] Further, step S1 includes: S11: Define the set of nodes in the distributed cluster , M This represents the total number of nodes in the distributed cluster. For the first M Each node; data is collected for each node within the sliding time window. T The internal state characteristics include network quality characteristics, computing power characteristics, and storage characteristics; S12: Normalize and positiveize the network quality characteristics, computing power characteristics, and storage characteristics to obtain normalized data of state characteristics; For negative features in the state characteristics, their normalized data are taken as follows: For positive features in the state features, their normalized data are taken as follows: ; in, i The number representing the state feature. For the first i Each state feature For the first i Normalized data of each state feature Sliding time window T Inner i The minimum and maximum values ​​of each state characteristic; S13: Construct a node feature matrix using normalized data of state features. The number of columns in the node feature matrix is ​​equal to the total number of state features. I The number of rows corresponds to the total number of nodes. M ; ; in, For the first M The node of the first I Normalized data of each state feature.

[0007] Further, step S2 includes: S21: Filtering the node feature matrix The eigenmaxima matrix is ​​obtained by finding the maximum value of each column element. ; ; in, for M The node in the node I The maximum value of the normalized data of each state feature; S22 uses the eigenmaxima matrix For reference, the comprehensive dynamic weight of each node is calculated to obtain the sliding time window. T Node weight matrix ; ; in, For the first M Each node in the sliding time window T The comprehensive dynamic weight; ; in, For the first M The node of the first i Normalized data of each state feature For the first M The node in the node i The weight coefficients of each state feature, and satisfying ; For the first M The combined dynamic weights of each node in the previous sliding time window. The length of the sliding time window. This is the time decay factor; S23: Calculate the node weight matrix The maximum value of the comprehensive dynamic weight The corresponding node serves as the global master node. When sliding time window T Global master node within Corresponding comprehensive dynamic weight At that time, the next sliding time window is triggered to re-elect the global master node. Otherwise, the next sliding time window will use the currently elected global master node. ; This represents the election weight coefficient for the global master node; S24: Obtain the comprehensive dynamic weight of all nodes in each region within the distributed cluster, obtain the comprehensive dynamic weight set of the regional nodes, select the maximum value from the comprehensive dynamic weight set, and the node corresponding to the maximum value is taken as the regional master node; S25: The global master node only communicates synchronously with the regional master nodes. The regional master nodes are responsible for the synchronous scheduling and consensus management of the sub-nodes in their respective regions. Data can be transmitted between the sub-nodes in the region, forming a three-layer cluster architecture of global master node - regional master node - regional sub-node.

[0008] Further, step S3 includes: S31: Define the collection of data objects within the distributed cluster , N For the number of data objects, For the first N Each data object Corresponding to a unique primary key identifier, n For the number of the data object; S32: Capture the set of operations for incremental transactions based on the transaction log of the elected global master node. , Q For the number of operations, For the first incremental transaction Q Each operation It includes information such as operation type, operation object, values ​​before and after the operation, transaction sequence number, and timestamp. q For operation number; S33: Operation set based on incremental transactions Get data object The change operation obtains incremental data, which is then used as a data object. The update content involves dividing the incremental data into fixed-size blocks, and then dividing the data objects... The updated content is divided into a set of data blocks. ; U The number of data blocks, For the first U One data block; S34: For each data block Calculate weak hash value With strong hash value ; S35: For a set of data blocks Each data block Construct a hash table with "Primary Key Identifier → Weak Hash Value, Strong Hash Value, Block Offset, Block Length", and set its size to [size missing]. L For a sliding window, a mapping table is used to store data blocks within the sliding window. Compared with historical benchmark data blocks Perform a match; Match the weak hash values ​​in the hash map table with the historical benchmark data blocks. The weak hash value in the corresponding base block hash index table is matched. If the match is successful, step S36 is executed. Otherwise, if there is no duplicate content in the sliding window, the sliding window slides back 1 byte, the weak hash value is updated, and the matching continues. S36: Data blocks that successfully match weak hash values Extract its strong hash value and compare it with the strong hash value of the byte stream in the current sliding window; If the strong hash values ​​match, then it is determined to be a data block. Exact match, record historical baseline data blocks Block offset in the corresponding base block hash index table and data blocks byte length Skip all bytes that are completely matched and proceed to the differential encoding stage; If the strong hash values ​​are inconsistent, it is determined to be a weak hash value collision. The sliding window slides one byte backward, the weak hash values ​​are updated, and the matching continues. S37: For a completely matched data block Perform differential encoding to convert the block offset and byte length Encode it into an incremental synchronization data packet, which is then used as a compressed incremental synchronization data packet.

[0009] Furthermore, weak hash values With strong hash value The calculation method is as follows: ; ; in, Representing data blocks The sum of all byte values ​​modulo 65521, and the data block. The result of a position-weighted sum of each byte modulo 65521; For data blocks byte length, For data blocks Inner i Decimal value of one byte. This indicates that a 256-bit hash value is output using a secure hash algorithm.

[0010] Further, step S4 includes: S41: Obtain two nodes within a three-tier cluster architecture m , v Average round-trip latency of data transmission paths between Available bandwidth Synchronizing with historical success rates Construct a synchronization cost model between nodes and calculate the synchronization cost between two nodes. m , v Synchronization cost coefficient between ; ; in, For the ideal average round-trip time, For an ideal historical synchronization success rate, The size of the incremental synchronization data packets; S42: Global Master Node As the root node, construct the initial set of synchronization tree nodes. excluding the global master node All regional master nodes and regional slave nodes are considered as the set of nodes not included in the synchronization tree. The optimal edge set of the synchronization tree is Optimal edge set Used to store the optimal set of paths for transmitting incremental synchronization data packets; S43: Using the root node as a reference, construct the objective function for the optimal path of transmitting incremental synchronization data packets using the synchronization cost model; ; in, As the regional master node, To form the optimal path with the regional master node, the regional secondary node, With regional secondary nodes The secondary nodes of the region that form the optimal path global master node With regional master node The synchronization cost coefficient between them regional master node With regional secondary nodes The synchronization cost coefficient between them This represents the synchronization cost coefficient between two regional secondary nodes. For the set of regional master nodes, For the set of secondary nodes in the region, This represents the optimal path formed between the secondary nodes of the region. S44: The global master node first obtains the incremental synchronization data packet, and then uses the optimal edge set as... The optimal path set in the system distributes incremental synchronization data packets to the regional master node and regional slave node, completing the incremental data synchronization of the distributed cluster.

[0011] A terminal device is provided, comprising a processor, a transceiver, and a memory. The memory stores a computer program, and the processor retrieves and runs the computer program from the memory, controlling the transceiver to perform receiving or sending actions, thereby enabling the terminal device to execute the aforementioned distributed data synchronization method.

[0012] A computer storage medium is provided for storing a computer program, the computer program including instructions for performing the distributed data synchronization method described above.

[0013] The beneficial effects of this invention are as follows: It achieves adaptive master node election and hierarchical cluster architecture construction through a multi-dimensional dynamic node weight quantization model, solving the performance bottleneck problem of traditional single-master architecture; it reduces incremental data transmission volume and optimizes cross-regional synchronization bandwidth usage through a sliding window block-level differential compression model. It realizes end-to-end quantization optimization of the distributed synchronization process, significantly improving synchronization throughput, reducing transmission latency and bandwidth costs while ensuring strong consistency. It also possesses strong heterogeneous cluster adaptability and fault tolerance capabilities, and can be widely applied in scenarios such as cross-regional distributed databases, cloud-native storage, and multi-center data disaster recovery. Attached Figure Description

[0014] Figure 1 This is a flowchart of a distributed data synchronization method. Detailed Implementation

[0015] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0016] like Figure 1 As shown, a distributed data synchronization method includes the following steps: S1: Collect multi-dimensional state features of all nodes in the distributed cluster within the sliding time window, and normalize and positiveize all state features to construct a node feature matrix.

[0017] Step S1 specifically includes the following steps: S11: Define the set of nodes in the distributed cluster , M This represents the total number of nodes in the distributed cluster. For the first M Each node; data is collected for each node within the sliding time window. T The internal state characteristics include network quality characteristics, computing power characteristics, and storage characteristics; The network quality characteristics in this embodiment include the average data transmission latency between nodes, the average packet loss rate during node data transmission, and the available bandwidth of the nodes.

[0018] The computing power characteristics of this embodiment include node CPU idle rate and memory idle rate; The storage features of this embodiment include the percentage of remaining storage space on the node disk and the random read / write I / O rate; S12: Normalize and positiveize the network quality characteristics, computing power characteristics, and storage characteristics to obtain normalized data of state characteristics; For negative features in the state characteristics, their normalized data are taken as follows: For positive features in the state features, their normalized data are taken as follows: In this embodiment, the negative features mainly include average data transmission latency and average packet loss rate.

[0019] in, i The number representing the state feature. For the first i Each state feature For the first i Normalized data of each state feature Sliding time window T Inner i The minimum and maximum values ​​of each state characteristic; S13: Construct a node feature matrix using normalized data of state features. The number of columns in the node feature matrix is ​​equal to the total number of state features. I The number of rows corresponds to the total number of nodes. M ; ; in, For the first M The node of the first I Normalized data of each state feature.

[0020] This invention proposes a dynamic weight quantification model for nodes that integrates network quality, computing power, storage, and historical reliability. A time decay factor ensures the smoothness of the weights, avoiding architectural oscillations caused by instantaneous index fluctuations. Based on weights, a hierarchical cluster architecture of a global master node and regional sub-master nodes is implemented. The global master node only needs to communicate with a few regional master nodes, significantly reducing the communication pressure on the master node globally. Simultaneously, a weight-based voting consensus mechanism is proposed, allowing high-performance, high-reliability nodes to have higher voting weights. The consensus process is no longer limited by weak nodes, solving the performance bottleneck problem of traditional synchronization schemes at the architectural level.

[0021] S2: Calculate the comprehensive dynamic weight of each node based on the node feature matrix, and use the comprehensive dynamic weight to elect the global master node and regional deputy master nodes. Combine the network topology to divide the distributed cluster into regions and construct a three-layer cluster architecture of global master node - regional master node - regional deputy node.

[0022] Step S2 specifically includes the following steps: S21: Filtering the node feature matrix The eigenmaxima matrix is ​​obtained by finding the maximum value of each column element. ; ; in, for M The node in the node I The maximum value of the normalized data of each state feature; S22 uses the eigenmaxima matrix For reference, the comprehensive dynamic weight of each node is calculated to obtain the sliding time window. T Node weight matrix ; ; in, For the first M Each node in the sliding time window T The comprehensive dynamic weight; ; in, For the first M The node of the first i Normalized data of each state feature For the first M The node in the node i The weight coefficients of each state feature, and satisfying ; For the first M The combined dynamic weights of each node in the previous sliding time window. The length of the sliding time window. This is the time decay factor; By integrating dynamic weights Characterization in sliding time window T Overall performance of internal nodes, including dynamic weights The larger the value, the better the performance, and vice versa.

[0023] S23: Calculate the node weight matrix The maximum value of the comprehensive dynamic weight The corresponding node serves as the global master node. When sliding time window TGlobal master node within Corresponding comprehensive dynamic weight At that time, the next sliding time window is triggered to re-elect the global master node. Otherwise, the next sliding time window will use the currently elected global master node. ; This is the election weight coefficient for the global master node, typically ranging from 0.7 to 0.9. S24: Obtain the comprehensive dynamic weight of all nodes in each region within the distributed cluster, obtain the comprehensive dynamic weight set of the regional nodes, select the maximum value from the comprehensive dynamic weight set, and the node corresponding to the maximum value is taken as the regional master node; S25: The global master node only communicates synchronously with the regional master nodes. The regional master nodes are responsible for the synchronous scheduling and consensus management of the sub-nodes in their respective regions. Data can be transmitted between the sub-nodes in the region, forming a three-layer cluster architecture of global master node - regional master node - regional sub-node.

[0024] S3: Based on the transaction log of the global master node, change data is captured and the operation set of incremental transactions is extracted; the incremental data is divided into fixed-size blocks, and the incremental data is compressed by strong and weak double hash matching and sliding window differential encoding to obtain compressed incremental synchronization data packets.

[0025] Step S3 specifically includes the following steps: S31: Define the collection of data objects within the distributed cluster , N For the number of data objects, For the first N Each data object Corresponding to a unique primary key identifier, n For the number of the data object; S32: Capture the set of operations for incremental transactions based on the transaction log of the elected global master node. , Q For the number of operations, For the first incremental transaction Q Each operation It includes information such as operation type, operation object, values ​​before and after the operation, transaction sequence number, and timestamp. q For operation number; S33: Operation set based on incremental transactions Get data object The change operation obtains incremental data, which is then used as a data object. The update content involves dividing the incremental data into fixed-size blocks, and then dividing the data objects... The updated content is divided into a set of data blocks. ; U The number of data blocks, For the first U One data block; S34: For each data block Calculate weak hash value With strong hash value ; ; ; in, Representing data blocks The sum of all byte values ​​modulo 65521, and the data block. The result of a position-weighted sum of each byte modulo 65521; For data blocks byte length, For data blocks Inner i Decimal value of one byte. This indicates that a 256-bit hash value is output using a secure hash algorithm; Weak hash value It uses 32-bit unsigned integers, is more than 10 times faster than MD5, supports sliding window calculation (when the window slides 1 byte, there is no need to recalculate the hash of the entire block; it only needs to be updated quickly through the previous S1 and S2 iterations), and has a strong hash value. The output is a 256-bit (32-byte) hash value, which has extremely strong collision resistance. Theoretically, the probability of two blocks with different content producing the same hash value is less than 10⁻⁷⁷, which can completely guarantee the accuracy of block content consistency verification.

[0026] S35: For a set of data blocks Each data block Construct a hash table with "Primary Key Identifier → Weak Hash Value, Strong Hash Value, Block Offset, Block Length", and set its size to [size missing]. L For a sliding window, a mapping table is used to store data blocks within the sliding window. Compared with historical benchmark data blocks Perform a match; Match the weak hash values ​​in the hash map table with the historical benchmark data blocks. The weak hash value in the corresponding base block hash index table is matched. If the match is successful, step S36 is executed. Otherwise, if there is no duplicate content in the sliding window, the sliding window slides back 1 byte, the weak hash value is updated, and the matching continues. S36: Data blocks that successfully match weak hash values Extract its strong hash value and compare it with the strong hash value of the byte stream in the current sliding window; If the strong hash values ​​match, then it is determined to be a data block. Exact match, record historical baseline data blocks Block offset in the corresponding base block hash index table and data blocks byte length Skip all bytes that are completely matched and proceed to the differential encoding stage; If the strong hash values ​​are inconsistent, it is determined to be a weak hash value collision. The sliding window slides one byte backward, the weak hash values ​​are updated, and the matching continues. S37: For a completely matched data block Perform differential encoding to convert the block offset and byte length Encode it into an incremental synchronization data packet, which is then used as a compressed incremental synchronization data packet.

[0027] Exact match: The encoded incremental synchronization data packet contains only two integer values, which are the offsets of the base block. (4 bytes), Matching byte length (4 bytes), totaling 8 bytes, far smaller than the standard block length of 16KB, achieving extreme compression.

[0028] Not a complete match: The encoded data is a data block. The complete binary content is provided for scenarios where the receiving end cannot reconstruct the data from the reference block, ensuring data integrity.

[0029] The sliding window scrolling matching mechanism breaks through the limitations of fixed block matching, enabling the matching of duplicate content across blocks and significantly improving the block matching success rate. Compared with the fixed block differential scheme, the compression rate can be improved by more than 30%. The matching process, which first uses weak hashing for fast filtering and then uses strong hashing for precise verification, minimizes the CPU overhead of matching calculations while ensuring 100% data accuracy, making it suitable for high-concurrency incremental data synchronization scenarios. The extremely simple differential encoding rules ensure that the receiving end can restore the incremental data block content 100% losslessly using the base block + offset + length, without any information loss, meeting the strong consistency requirements of distributed data synchronization.

[0030] S4: Based on the three-layer cluster architecture and the comprehensive dynamic weight of nodes, a synchronization cost model between nodes is constructed. The minimum synchronization cost spanning tree is generated with the global master node as the root node to obtain the optimal synchronization path. Incremental synchronization data packets are transmitted to the regional master node and regional slave node using the optimal synchronization path to complete the incremental data synchronization of the distributed cluster.

[0031] Step S4 specifically includes the following steps: S41: Obtain two nodes within a three-tier cluster architecture m , v Average round-trip latency of data transmission paths between Available bandwidth Synchronizing with historical success rates Construct a synchronization cost model between nodes and calculate the synchronization cost between two nodes. m , v Synchronization cost coefficient between ; ; in, For the ideal average round-trip time, For an ideal historical synchronization success rate, The size of the incremental synchronization data packets; Synchronization cost coefficient Synchronization cost coefficient is used to characterize the data transmission quality of the transmission path between nodes. The larger the value, the better the data transmission quality; conversely, the smaller the value, the worse the data transmission quality.

[0032] S42: Global Master Node As the root node, construct the initial set of synchronization tree nodes. excluding the global master node All regional master nodes and regional slave nodes are considered as the set of nodes not included in the synchronization tree. The optimal edge set of the synchronization tree is Optimal edge set Used to store the optimal set of paths for transmitting incremental synchronization data packets; S43: Using the root node as a reference, construct the objective function for the optimal path of transmitting incremental synchronization data packets using the synchronization cost model; ; in, As the regional master node, To form the optimal path with the regional master node, the regional secondary node, With regional secondary nodes The secondary nodes of the region that form the optimal path global master node With regional master node The synchronization cost coefficient between them regional master node With regional secondary nodes The synchronization cost coefficient between them This represents the synchronization cost coefficient between two regional secondary nodes. For the set of regional master nodes, For the set of secondary nodes in the region, This represents the optimal path formed between the secondary nodes of the region. S44: The global master node first obtains the incremental synchronization data packet, and then uses the optimal edge set as... The optimal path set in the system distributes incremental synchronization data packets to the regional master node and regional slave node, completing the incremental data synchronization of the distributed cluster.

[0033] Since the global master node only communicates synchronously with the regional master nodes, and the regional master nodes are responsible for the synchronization scheduling and consensus management of the secondary nodes within their respective regions, and data transmission can occur between secondary nodes within a region, a fixed path is formed between the global master node and the regional master nodes. Therefore, their synchronization cost coefficient is... It is fixed; the objective function needs to select one or more regions to receive data from the region's master node, which is actually based on... Path optimization ensures that the globally optimal data synchronization path is obtained.

[0034] This invention addresses the pain points of incremental data transmission by proposing a sliding window block-level differential compression model based on strong and weak dual hashing. Weak hashing enables fast block matching, while strong hashing ensures accurate consistency verification. Only incrementally changing block data is transmitted, or even just the offset and length of matching blocks, significantly reducing the amount of data transmitted. Simultaneously, a synchronization cost model between nodes is constructed, generating a minimum synchronization cost spanning tree rooted at the master node. This replaces traditional broadcast synchronization, achieving global optimization of the synchronization path and significantly reducing bandwidth consumption and transmission latency in cross-regional synchronization.

[0035] A terminal device includes a processor, a transceiver, and a memory. The memory stores a computer program, and the processor retrieves and runs the computer program from the memory, controlling the transceiver to perform receiving or sending actions, thereby enabling the terminal device to execute the aforementioned distributed data synchronization method.

[0036] A computer storage medium for storing a computer program, the computer program including instructions for performing the distributed data synchronization method described above.

[0037] This system achieves adaptive master node election and hierarchical cluster architecture construction through a multi-dimensional dynamic node weight quantization model, solving the performance bottleneck problem of traditional single-master architecture. It also reduces incremental data transmission volume and optimizes cross-regional synchronization bandwidth usage through a sliding window block-level differential compression model. This system realizes end-to-end quantization optimization of the distributed synchronization process, significantly improving synchronization throughput, reducing transmission latency and bandwidth costs while ensuring strong consistency. Furthermore, it possesses strong heterogeneous cluster adaptability and fault tolerance capabilities, making it widely applicable to scenarios such as cross-regional distributed databases, cloud-native storage, and multi-center data disaster recovery.

Claims

1. A distributed data synchronization method, characterized in that, Includes the following steps: S1: Collect multi-dimensional state features of all nodes in the distributed cluster within the sliding time window, and normalize and positiveize all state features to construct a node feature matrix. S2: Calculate the comprehensive dynamic weight of each node based on the node feature matrix, and use the comprehensive dynamic weight to elect the global master node and regional deputy master node. Combine the network topology to divide the distributed cluster into regions and construct a three-layer cluster architecture of global master node - regional master node - regional deputy node. S3: Based on the transaction log of the global master node, change data is captured and the operation set of incremental transactions is extracted; the incremental data is divided into fixed-size blocks, and the incremental data is compressed by strong and weak double hash matching and sliding window differential encoding to obtain compressed incremental synchronization data packets. S4: Based on the three-layer cluster architecture and the comprehensive dynamic weight of nodes, a synchronization cost model between nodes is constructed. The minimum synchronization cost spanning tree is generated with the global master node as the root node to obtain the optimal synchronization path. Incremental synchronization data packets are transmitted to the regional master node and regional slave node using the optimal synchronization path to complete the incremental data synchronization of the distributed cluster.

2. The distributed data synchronization method according to claim 1, characterized in that, Step S1 includes: S11: Define the set of nodes in the distributed cluster , M This represents the total number of nodes in the distributed cluster. For the first M Each node; data is collected for each node within the sliding time window. T The internal state characteristics include network quality characteristics, computing power characteristics, and storage characteristics; S12: Normalize and positiveize the network quality characteristics, computing power characteristics, and storage characteristics to obtain normalized data of state characteristics; For negative features in the state characteristics, their normalized data are taken as follows: For positive features in the state features, their normalized data are taken as follows: ; in, i The number representing the state feature. For the first i Each state feature For the first i Normalized data of each state feature Sliding time window T Inner i The minimum and maximum values ​​of each state characteristic; S13: Construct a node feature matrix using normalized data of state features. The number of columns in the node feature matrix is ​​equal to the total number of state features. I The number of rows corresponds to the total number of nodes. M ; ； in, For the first M The node of the first I Normalized data of each state feature.

3. The distributed data synchronization method according to claim 2, characterized in that, Step S2 includes: S21: Filtering the node feature matrix The eigenmaxima matrix is ​​obtained by finding the maximum value of each column element. ; ； in, for M The node in the node I The maximum value of the normalized data of each state feature; S22 uses the eigenmaxima matrix For reference, the comprehensive dynamic weight of each node is calculated to obtain the sliding time window. T Node weight matrix ; ； in, For the first M Each node in the sliding time window T The comprehensive dynamic weight; ； in, For the first M The node of the first i Normalized data of each state feature For the first M The node in the node i The weight coefficients of each state feature, and satisfying ; For the first M The combined dynamic weights of each node in the previous sliding time window. The length of the sliding time window. This is the time decay factor; S23: Calculate the node weight matrix The maximum value of the comprehensive dynamic weight The corresponding node serves as the global master node. When sliding time window T Global master node within Corresponding comprehensive dynamic weight At that time, the next sliding time window is triggered to re-elect the global master node. Otherwise, the next sliding time window will use the currently elected global master node. ; This represents the election weight coefficient for the global master node; S24: Obtain the comprehensive dynamic weight of all nodes in each region within the distributed cluster, obtain the comprehensive dynamic weight set of the regional nodes, select the maximum value from the comprehensive dynamic weight set, and the node corresponding to the maximum value is taken as the regional master node; S25: The global master node only communicates synchronously with the regional master nodes. The regional master nodes are responsible for the synchronous scheduling and consensus management of the sub-nodes in their respective regions. Data can be transmitted between the sub-nodes in the region, forming a three-layer cluster architecture of global master node - regional master node - regional sub-node.

4. The distributed data synchronization method according to claim 3, characterized in that, Step S3 includes: S31: Define the collection of data objects within the distributed cluster , N For the number of data objects, For the first N Each data object Corresponding to a unique primary key identifier, n For the number of the data object; S32: Capture the set of operations for incremental transactions based on the transaction log of the elected global master node. , Q For the number of operations, For the first incremental transaction Q Each operation It includes information such as operation type, operation object, values ​​before and after the operation, transaction sequence number, and timestamp. q For operation number; S33: Operation set based on incremental transactions Get data object The change operation obtains incremental data, which is then used as a data object. The update content involves dividing the incremental data into fixed-size blocks, and then dividing the data objects... The updated content is divided into a set of data blocks. ; U The number of data blocks, For the first U One data block; S34: For each data block Calculate weak hash value With strong hash value ; S35: For a set of data blocks Each data block Construct a hash table with "Primary Key Identifier → Weak Hash Value, Strong Hash Value, Block Offset, Block Length", and set its size to [size missing]. L For a sliding window, a mapping table is used to store data blocks within the sliding window. Compared with historical benchmark data blocks Perform a match; Match the weak hash values ​​in the hash map table with the historical benchmark data blocks. The weak hash value in the corresponding base block hash index table is matched. If the match is successful, step S36 is executed. Otherwise, if there is no duplicate content in the sliding window, the sliding window slides back 1 byte, the weak hash value is updated, and the matching continues. S36: Data blocks that successfully match weak hash values Extract its strong hash value and compare it with the strong hash value of the byte stream in the current sliding window; If the strong hash values ​​match, then it is determined to be a data block. Exact match, record historical baseline data blocks Block offset in the corresponding base block hash index table and data blocks byte length Skip all bytes that are completely matched and proceed to the differential encoding stage; If the strong hash values ​​are inconsistent, it is determined to be a weak hash value collision. The sliding window slides one byte backward, the weak hash values ​​are updated, and the matching continues. S37: For a completely matched data block Perform differential encoding to convert the block offset and byte length Encode it into an incremental synchronization data packet, which is then used as a compressed incremental synchronization data packet.

5. The distributed data synchronization method according to claim 4, characterized in that, The weak hash value With strong hash value The calculation method is as follows: ； ； in, Representing data blocks The sum of all byte values ​​modulo 65521, and the data block. The result of a position-weighted sum of each byte modulo 65521; For data blocks byte length, For data blocks Inner i Decimal value of one byte, This indicates that a 256-bit hash value is output using a secure hash algorithm.

6. The distributed data synchronization method according to claim 4, characterized in that, Step S4 includes: S41: Obtain two nodes within a three-tier cluster architecture m , v Average round-trip latency of data transmission paths between Available bandwidth Synchronizing success rate with history Construct a synchronization cost model between nodes and calculate the synchronization cost between two nodes. m , v Synchronization cost coefficient between ; ； in, For the ideal average round-trip time, For an ideal historical synchronization success rate, The size of the incremental synchronization data packets; S42: Global Master Node As the root node, construct the initial set of synchronization tree nodes. , except for the global master node All regional master nodes and regional slave nodes are considered as the set of nodes not included in the synchronization tree. The optimal edge set of the synchronization tree is Optimal edge set Used to store the optimal set of paths for transmitting incremental synchronization data packets; S43: Using the root node as a reference, construct the objective function for the optimal path of transmitting incremental synchronization data packets using the synchronization cost model; ； in, As the regional master node, To form the optimal path with the regional master node, the regional secondary node, With regional secondary nodes The secondary nodes of the region that form the optimal path global master node With regional master node The synchronization cost coefficient between them regional master node With regional secondary nodes The synchronization cost coefficient between them This represents the synchronization cost coefficient between two regional secondary nodes. For the set of regional master nodes, For the set of secondary nodes in the region, This represents the optimal path formed between the secondary nodes of the region. S44: The global master node first obtains the incremental synchronization data packet, and then uses the optimal edge set as... The optimal path set in the system distributes incremental synchronization data packets to the regional master node and regional slave node, completing the incremental data synchronization of the distributed cluster.

7. A terminal device, characterized in that, The device includes a processor, a transceiver, and a memory. The memory stores a computer program, and the processor retrieves and runs the computer program from the memory. The processor controls the transceiver to perform receiving or sending actions, causing the terminal device to execute the distributed data synchronization method as described in any one of claims 1-6.

8. A computer storage medium, characterized in that, Used to store computer programs, the computer programs including instructions for performing the distributed data synchronization method as described in any one of claims 1-5.

Images