Data deduplication method for cloud backup platform based on distributed storage

Abstract

The present invention relates to the field of data processing technology, and in particular to a data deduplication method for a cloud backup platform based on distributed storage. The method comprises the following steps: using a dynamic block algorithm to adaptively block data, generate a unique identification code and extract multi-dimensional feature information; establishing a global index through a distributed hash table, mapping the data location and maintaining index consistency; performing redundancy analysis and similarity matching based on a deep learning algorithm, and executing a deduplication strategy; using a hierarchical graph partitioning algorithm to achieve distributed storage of data blocks according to access frequency and load conditions; and using a reinforcement learning algorithm to dynamically optimize the block strategy, index frequency and storage layout to achieve global optimization of performance and resources. The present invention achieves high efficiency, accuracy and dynamic optimization of data deduplication, and improves the data storage and management efficiency of the cloud backup platform.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data deduplication method for a cloud backup platform based on distributed storage. Background Art

[0002] With the rapid development of cloud computing and big data technologies, more and more businesses and individuals are migrating data storage and backup to cloud platforms to ensure data security and improve data management efficiency. As an important cloud storage application, cloud backup platforms provide users with remote data backup and recovery services. However, with the rapid growth of data volumes, cloud backup platforms face numerous challenges, particularly data redundancy.

[0003] In cloud backup scenarios, user data backups often contain large amounts of duplicate data. For example, for different versions of the same file, only part of the content changes, while the majority of the content remains the same; or for files uploaded by multiple users, duplicate data may exist due to the same data source. The existence of this duplicate data takes up a large amount of storage space, increasing storage costs, while also consuming more network bandwidth and computing resources, affecting the performance and efficiency of the cloud backup platform. Currently, the following problems also exist: Existing technologies often have difficulty effectively ensuring data consistency when establishing and maintaining global indexes, which can easily lead to index failure or storage location mismatches, affecting data location and deduplication performance; traditional data deduplication methods mainly rely on simple hash matching, which cannot conduct in-depth analysis of data block characteristics, making it difficult to identify similar data blocks, and failing to fully tap the potential of data redundancy, resulting in low storage space utilization; existing technologies lack effective storage optimization strategies when storing data after deduplication, resulting in uneven data distribution and excessive load on some storage nodes, affecting system performance and stability. Summary of the Invention

[0004] To solve the above problems, the present invention provides a data deduplication method for a cloud backup platform based on distributed storage, which solves the problem of how to effectively solve the data redundancy problem in the cloud backup platform, ensures data consistency and deduplication performance, and improves the accuracy of data deduplication by identifying similar data blocks through in-depth analysis. At the same time, it optimizes data storage distribution and reduces the problem of uneven load on storage nodes, thereby achieving high efficiency, accuracy and dynamic optimization of data deduplication, and improving the data storage and management efficiency of the cloud backup platform.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The data deduplication method of a cloud backup platform based on distributed storage includes the following steps:

[0007] S1: Adaptively partitions backup data using a dynamic partitioning algorithm, generates a unique identification code for each partition, and extracts multi-dimensional feature information based on an intelligent feature extraction algorithm, including content distribution features, redundancy pattern features, and compression potential assessment features.

[0008] S2: A global index is established through a distributed hash table, mapping the unique identifier of each block to its storage location, and a distributed consistency protocol is used to dynamically maintain the consistency of the index;

[0009] S3: Based on the global index, a deep learning algorithm is used to perform redundancy analysis and similarity matching on the block features, identify duplicate or similar data blocks globally, and implement a deduplication strategy based on the detection results;

[0010] S4: Based on access frequency and load, a layered graph partitioning algorithm is used to map the deduplicated data blocks into a cross-node storage graph model, and the deduplicated data blocks are distributed and stored on multiple nodes.

[0011] S5: Based on reinforcement learning algorithms, it monitors the performance of the deduplication process in real time and dynamically optimizes the block strategy, indexing frequency, and storage distribution to achieve global optimization of performance and resource utilization.

[0012] Furthermore, the construction process of the dynamic block algorithm includes the following steps:

[0013] Based on the content characteristics of the data to be backed up, the sliding window technology is combined with the sensitivity adjustment mechanism to dynamically segment the data and generate an initial block set with boundary adaptability.

[0014] The initial block set is divided through a multi-level block optimization strategy, combined with the data content change rate and block granularity adjustment factor;

[0015] Generate a unique identification code for each block, where the identification code is generated based on a cryptographic hash algorithm and a joint calculation of the block characteristics;

[0016] After the block segmentation is completed, the characteristics of each block are evaluated, and the block segmentation strategy is dynamically adjusted based on the block stability index and specific content distribution rules.

[0017] Furthermore, step S3 includes the following steps:

[0018] Obtain multi-dimensional feature information of the blocks based on the global index, and generate high-dimensional feature vectors using feature embedding algorithms;

[0019] Adaptive variational autoencoders are used to reduce the dimensionality of high-dimensional feature vectors to generate a latent feature space representation. Density clustering analysis is used to evaluate the similarity and repetitiveness between blocks and to construct a similarity matrix of block features.

[0020] Based on the similarity matrix, a distributed graph neural network is used to dynamically construct a block feature graph, perform redundancy detection and duplicate clustering analysis on the blocks, and improve the efficiency and accuracy of duplicate detection through a cross-node distributed collaboration mechanism;

[0021] According to the duplicate detection results, the edge entropy optimal deduplication algorithm is used to perform deduplication operations on duplicate or similar data blocks, delete completely duplicate blocks, and implement a merge storage strategy for similar blocks.

[0022] Furthermore, the formula of the distributed graph neural network is as follows:

[0023]

[0024] in, Represents the feature vector of block v after being updated by the graph neural network; and Represents the multidimensional feature data of data blocks; F u and F v represents the additional feature vectors of nodes u and v, including relevant metadata obtained from the global index; N(v) represents the set of neighbor nodes of node v; d u and d v Indicates the number of other similar blocks it is connected to; represents the attention weight, that is, the similarity between node u and node v; W (l) Represents the trainable weight matrix of the graph neural network at layer l; represents the adaptive weight coefficient; σ represents the activation function.

[0025] Furthermore, step S4 includes the following steps:

[0026] Based on the deduplicated data blocks, the importance of the data blocks is dynamically evaluated using access frequency and load balancing strategies to calculate the storage priority of each data block.

[0027] Based on the characteristics of the deduplicated data blocks and the node load status, a cross-node storage graph model is constructed using a layered graph partitioning algorithm. In the graph, nodes represent storage nodes, and edges represent the transmission path weights between nodes.

[0028] Based on the constructed cross-node storage graph model, the graph optimization algorithm is used to generate the optimal block storage distribution solution;

[0029] Through the reinforcement learning algorithm, the node storage status and the distribution status of the access blocks are monitored in real time, and the distribution storage strategy of the deduplicated data blocks is dynamically adjusted.

[0030] Furthermore, the formula of the layered graph partitioning algorithm is as follows:

[0031]

[0032] Where min F represents the objective function of hierarchical graph partitioning, which is used to describe the optimization goal of data block storage distribution; N represents the total number of data blocks; M represents the total number of storage nodes; represents the set of neighboring data blocks of data block i; w ij represents the transmission weight between data block i and data block j; d ij represents the network delay or transmission path weight between different nodes; f ij represents the similarity characteristics of data blocks i and j; c represents the weight coefficient; L k represents the current load of storage node k; C k Represents the capacity of storage node k.

[0033] Furthermore, step S4 also includes storing the data blocks determined to be duplicates only once, and managing other duplicate references through logical pointers; for newly added non-duplicate data blocks, efficient storage is achieved by combining compression algorithms and hot and cold data tiering strategies, while introducing an incremental data flow model to dynamically optimize the storage layout.

[0034] Furthermore, the establishment of the global index supports efficient associative mapping between high-dimensional features and physical storage locations through a multi-dimensional index mapping mechanism combined with a content-aware hash algorithm based on block features.

[0035] Furthermore, the reinforcement learning algorithm is based on a two-layer hierarchical optimization framework. The top layer uses a policy gradient method to optimize the global performance objectives of the deduplication process, and the bottom layer combines a multi-agent collaboration mechanism to achieve coordinated dynamic adjustment of the block strategy, indexing frequency and storage mapping.

[0036] The beneficial effects of the present invention are:

[0037] The present invention adaptively blocks the backup data through a dynamic block algorithm, and extracts multidimensional feature information (such as content distribution features, redundancy pattern features and compression potential assessment features) in combination with an intelligent feature extraction algorithm. It can accurately locate the redundancy and similarity of data, effectively improving the accuracy and efficiency of data deduplication. A global index is established through a distributed hash table, and a distributed consistency protocol is used to dynamically maintain the consistency of the index, ensuring that the unique identification of the data block and the storage location mapping are accurate, while ensuring the high availability and reliability of the index in the distributed system. A deep learning algorithm is used to perform redundancy analysis and similarity matching on the features of the data blocks, which can efficiently identify duplicate or similar data blocks on a global scale, realize intelligent data deduplication, and effectively reduce storage space occupancy. The deduplicated data blocks are mapped to a cross-node storage graph model through a hierarchical graph partitioning algorithm, and distributed storage is performed in combination with data access frequency and node load conditions, further realizing data load balancing and improving the utilization of storage resources. Based on the reinforcement learning algorithm, the performance of the deduplication process is monitored in real time, and the block strategy, index update frequency and storage distribution are dynamically optimized to ensure that the data deduplication process reaches the global optimum in terms of performance and resource utilization, effectively improving the response speed and operating efficiency of the system. Through precise data deduplication and distributed storage mechanisms, the amount of duplicate data storage is significantly reduced, saving storage costs, while increasing the speed of data backup and recovery, and improving the overall performance and user experience of the cloud backup platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a data deduplication method for a cloud backup platform based on distributed storage according to the present invention.

[0039] Figure 2 It is a flowchart of step S3 provided by one embodiment of the present invention.

[0040] Figure 3 It is a flowchart of step S4 provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] See also Figure 1-3 As shown, the present invention relates to a data deduplication method for a cloud backup platform based on distributed storage.

[0042] Example

[0043] The data deduplication method of a cloud backup platform based on distributed storage includes the following steps:

[0044] S1: Adaptively partitions backup data using a dynamic partitioning algorithm, generates a unique identification code for each partition, and extracts multi-dimensional feature information based on an intelligent feature extraction algorithm, including content distribution features, redundancy pattern features, and compression potential assessment features.

[0045] The construction process of the dynamic block algorithm includes the following steps:

[0046] Based on the content characteristics of the data to be backed up, the sliding window technology is combined with the sensitivity adjustment mechanism to dynamically segment the data and generate an initial block set with boundary adaptability.

[0047] The initial block set is divided through a multi-level block optimization strategy, combined with the data content change rate and block granularity adjustment factor;

[0048] Specifically, the data content change rate of the initial block set is evaluated, and the degree of content change within the block is calculated (for example, through indicators such as data entropy and repetition rate).

[0049] Dynamically adjust the block size based on the change rate results:

[0050] High rate of change: Refine the block granularity and divide the data into smaller blocks;

[0051] Low change rate: Merge adjacent blocks to generate larger blocks, reducing the number of blocks and indexing burden.

[0052] By setting a block granularity range (e.g., minimum 4KB, maximum 64KB) and introducing a block granularity adjustment factor, the blocks are dynamically optimized based on data characteristics. For example, in a network transmission environment, bandwidth-sensitive data streams can be assigned a smaller block granularity to facilitate subsequent deduplication processing; in large file storage scenarios, the block granularity can be appropriately increased to improve storage efficiency. Through a multi-level block optimization strategy, the initial block set is re-divided into block sets with greater content adaptability and finer granularity.

[0053] Generate a unique identification code for each block, where the identification code is generated based on a cryptographic hash algorithm and a joint calculation of the block characteristics;

[0054] Specifically, for each final generated block, a cryptographic hash algorithm (such as SHA-256 or MD5) is used to calculate the hash value of the block. In order to improve the accuracy of the unique identification code and the distinguishability of the blocks, the hash result is jointly calculated in combination with the block characteristics (such as content distribution and change rate) to ensure uniqueness. The hash value of the block is combined with the content feature vector to generate a composite identification code. The generated unique identification code and the metadata corresponding to the block (such as block size, starting position, etc.) are recorded in the index table to provide a basis for subsequent data deduplication and storage mapping.

[0055] After the block segmentation is completed, the characteristics of each block are evaluated, and the block segmentation strategy is dynamically adjusted based on the block stability index and specific content distribution rules.

[0056] It should be noted that the stability of each block is evaluated mainly based on the following indicators:

[0057] If a block remains stable across different versions, it is considered a high-stability block;

[0058] The degree of repetitiveness of chunk boundaries: Frequent changes in boundaries may indicate that the chunking strategy needs to be optimized.

[0059] Analyze the data content distribution of the blocks and identify specific types of data (such as duplicate data blocks and data blocks with high compression potential).

[0060] Dynamically adjust the chunking strategy based on content distribution patterns: further refine the chunking of redundant data and pre-mark compressible data blocks to facilitate subsequent compression. Combining stability indicators and content pattern analysis, the system dynamically optimizes chunking parameters such as sensitivity and granularity adjustment factors, gradually improving chunking quality and efficiency.

[0061] S2: A global index is established through a distributed hash table, the unique identifier of each block is mapped to its storage location, and a distributed consistency protocol is used to dynamically maintain the consistency of the index; the establishment of the global index supports efficient associative mapping of high-dimensional features and physical storage locations through a multi-dimensional index mapping mechanism combined with a content-aware hash algorithm based on block features.

[0062] Specifically, a consistent hashing algorithm (such as Consistent Hashing) is used to establish a DHT ring structure, arranging storage nodes according to the hash value space. For example, a unique node ID is assigned to each storage node and mapped to a position in the logical ring using a hash function (such as SHA-256).

[0063] The data block's unique identifier (e.g., SHA-256) is also calculated using the same hash function and stored on the corresponding node based on its hash value. Each data block's unique identifier serves as the key, and the storage location (node address, offset, etc.) serves as the value, forming a key-value pair of <unique identifier, storage location>. If a data block's hash value conflicts with multiple nodes, a virtual node mechanism is used to evenly distribute the data block across different nodes. To prevent uneven storage node load, virtual node technology is used to split a physical node into multiple logical virtual nodes, improving the balance of data block indexing.

[0064] Extracted data block features (such as content distribution, redundancy patterns, and compression potential) are compressed into a low-dimensional space using dimensionality reduction algorithms (such as principal component analysis (PCA) and t-SNE). Content-aware hashing algorithms (such as SimHash and MinHash) are used to generate feature-dependent hash codes, ensuring that similar data blocks are mapped to similar locations, accelerating similarity retrieval.

[0065] Design a two-layer index structure:

[0066] Primary index: The unique identifier and storage node address of the data block stored through DHT.

[0067] Auxiliary index: stores the feature hash code of the data block, supporting feature retrieval and similarity matching.

[0068] A distributed hash search algorithm is used for the primary index, with a query complexity of O(log N). For the auxiliary index, locality-sensitive hashing (LSH) is used to achieve fast approximate search for similar features and reduce search overhead.

[0069] Use the Raft protocol or Paxos protocol to maintain the consistency of index data, ensuring that the index can be correctly synchronized when a node fails or a new node is added.

[0070] The Raft protocol works as follows: All nodes elect a leader; the leader is responsible for receiving index update requests and synchronizing the updates to other nodes (followers); if the leader fails, the system automatically re-elects the leader to ensure uninterrupted consistency maintenance.

[0071] When a data block is added, deleted, or modified, the node sends an index update request to the master node. The master node is responsible for executing the update and synchronizing the update to all replica nodes through the consistency protocol. An index version control mechanism is implemented to avoid index conflicts between different nodes.

[0072] To improve system reliability, index data is replicated to multiple nodes, forming a replication mechanism (such as triple-copy storage). If a node fails, the index data can be restored from the replica node to ensure uninterrupted indexing service.

[0073] When a new node joins the system, the system automatically migrates the indexes of some data blocks from the existing node to the new node, redistributing the index data and ensuring a balanced index load. When a node leaves the system, the indexes of the data blocks are automatically migrated to other available nodes to maintain the integrity of the index data. Based on the storage node load status and data access frequency, the system periodically executes a data redistribution algorithm to dynamically rebalance and optimize the global index.

[0074] S3: Based on the global index, a deep learning algorithm is used to perform redundancy analysis and similarity matching on the block features, identify duplicate or similar data blocks globally, and implement a deduplication strategy based on the detection results;

[0075] Wherein, the step S3 includes the following steps:

[0076] Obtain multi-dimensional feature information of the blocks based on the global index, and generate high-dimensional feature vectors using feature embedding algorithms;

[0077] Adaptive variational autoencoders are used to reduce the dimensionality of high-dimensional feature vectors to generate a latent feature space representation. Density clustering analysis is used to evaluate the similarity and repetitiveness between blocks and to construct a similarity matrix of block features.

[0078] Specifically, the high-dimensional feature vector is input into an adaptive variational autoencoder (VAE) to perform dimensionality reduction on the feature space.

[0079] The VAE model works by mapping the high-dimensional input feature vector to a low-dimensional latent feature space to extract the core features of the data block. The encoder dimension is dynamically adjusted based on the complexity of the features, ensuring that the resulting dimensionality reduction accurately preserves the representativeness of the features. Reconstructing the low-dimensional feature vector ensures information preservation during the dimensionality reduction process. The output is a representation vector for each block in the low-dimensional latent space, which more efficiently represents the redundancy and similarity of the blocks.

[0080] Use a density clustering algorithm (such as DBSCAN or HDBSCAN) to cluster latent feature vectors and assess the similarity and repetition between blocks. High-density regions indicate blocks with high similarity or duplicate content, while outliers indicate unique, non-duplicate blocks. The output is a similarity matrix between blocks, which is used for subsequent redundancy detection and cluster analysis.

[0081] Based on the similarity matrix, a distributed graph neural network is used to dynamically construct a block feature graph, perform redundancy detection and duplicate clustering analysis on the blocks, and improve the efficiency and accuracy of duplicate detection through a cross-node distributed collaboration mechanism;

[0082] Specifically, based on the similarity matrix, a distributed graph neural network (such as GraphSAGE and GAT) is used to dynamically construct a block feature graph:

[0083] Node: feature vector of each data block;

[0084] Edge: Similarity between nodes (edge weight).

[0085] Graph neural networks learn through node representation and further extract the relationship features between nodes.

[0086] Complete duplicate blocking: Nodes with higher edge weights in the graph are marked as complete duplicates through clustering;

[0087] Similar blocks: Nodes with high edge weights but not identical are identified as similar blocks through similarity matching.

[0088] In a distributed storage environment, the computational tasks of graph neural networks are distributed and collaboratively executed through a cross-node communication mechanism to improve the efficiency and accuracy of duplicate detection: each node is responsible for calculating the feature map of the local block; the intermediate results are shared with other nodes through a decentralized algorithm (such as the Gossip protocol) to construct a global block feature map.

[0089] According to the duplicate detection results, the edge entropy optimal deduplication algorithm is used to perform deduplication operations on duplicate or similar data blocks, delete completely duplicate blocks, and implement a merge storage strategy for similar blocks.

[0090] Specifically, the edge entropy optimal deduplication algorithm is used to perform deduplication operations on the detected duplicate or similar blocks:

[0091] Completely duplicate chunks: directly delete redundant chunks and keep only one physical copy; mark the storage locations of other duplicate chunks through logical references (such as pointers or index mappings).

[0092] Similar blocks: Use content difference merging technology (such as incremental storage method) to store only the differences between similar blocks, minimizing storage space usage.

[0093] Based on the deduplication results, the storage information after deleting redundant data is updated to the global index to ensure consistency between the index and storage layout. The cross-node storage distribution is updated to ensure load balancing and efficient access to data storage.

[0094] Furthermore, the formula of the distributed graph neural network is as follows:

[0095]

[0096] in, Represents the feature vector of block v after being updated by the graph neural network; and Represents the multidimensional feature data of data blocks; F u and F v represents the additional feature vectors of nodes u and v, including relevant metadata obtained from the global index; N(v) represents the set of neighbor nodes of node v; d u and d v Indicates the number of other similar blocks it is connected to; represents the attention weight, that is, the similarity between node u and node v; W (l) Represents the trainable weight matrix of the graph neural network at layer l; represents the adaptive weight coefficient; σ represents the activation function.

[0097] S4: Based on the access frequency and load conditions, a layered graph partitioning algorithm is used to map the deduplicated data blocks into a cross-node storage graph model, and the deduplicated data blocks are distributed and stored on multiple nodes; step S4 also includes storing the data blocks determined to be duplicates only once, and managing other duplicate references through logical pointers; for newly added non-duplicate data blocks, compression algorithms and hot and cold data tiering strategies are combined for efficient storage, and an incremental data flow model is introduced to dynamically optimize the storage layout.

[0098] It should be noted that a compression algorithm (such as LZ4, ZSTD) is executed on the newly added non-duplicate data blocks to reduce the storage occupancy of the data blocks.

[0099] Hot and cold tiered storage strategy:

[0100] Hot data: Highly accessed data blocks are stored in high-performance tiers (such as SSDs or NVMe storage nodes). This ensures minimal access latency and improves data access efficiency.

[0101] Cold data: Infrequently accessed data blocks are stored in high-capacity tiers (such as HDD storage nodes). Data blocks are compressed during storage to maximize storage space savings.

[0102] During the data backup and storage process, continuously monitor new or changed data blocks and build incremental data streams.

[0103] Incremental data is compared with stored data in real time to identify newly added data blocks and dynamically adjust the storage layout.

[0104] When the storage node load is unbalanced or storage space is insufficient, the system dynamically migrates data blocks:

[0105] Migrate low-priority data blocks to nodes with lower load or better storage capacity; retain high-priority data blocks on high-performance nodes. Maintain data accessibility during the migration process to avoid affecting system performance.

[0106] Wherein, the step S4 includes the following steps:

[0107] Based on the deduplicated data blocks, the importance of the data blocks is dynamically evaluated using access frequency and load balancing strategies to calculate the storage priority of each data block.

[0108] Specifically, we monitor the access frequency of the deduplicated data blocks in real time and collect data block access logs over a period of time. We use the sliding time window method to count the number of accesses to each data block:

[0109] High-frequency data blocks: These blocks are accessed frequently recently and are considered hot data.

[0110] Low-frequency data blocks: These blocks are accessed less frequently recently and are considered cold data.

[0111] The system monitors the storage occupancy, bandwidth usage, and load status of each storage node in real time. It comprehensively evaluates node load and generates a node load score: nodes with low loads have higher storage availability; nodes with high loads are prioritized for storage allocation.

[0112] Taking into account the access frequency of data blocks and node load conditions, the storage priority is dynamically calculated for each data block: high-frequency data blocks are preferentially stored in low-latency, high-bandwidth storage nodes; low-frequency data blocks are compressed and stored in large-capacity cold data storage nodes.

[0113] Based on the characteristics of the deduplicated data blocks and the node load status, a cross-node storage graph model is constructed using a layered graph partitioning algorithm. In the graph, nodes represent storage nodes, and edges represent the transmission path weights between nodes.

[0114] Specifically, in the definition of the graph model: the nodes in the graph represent storage nodes; the edges represent the transmission paths between nodes, and the weights are set according to indicators such as transmission bandwidth, node load, and access delay.

[0115] Each storage node reports its own status information in real time, including: storage space usage; current network bandwidth load; I / O processing capacity and access latency.

[0116] Using a hierarchical graph partitioning algorithm (such as METIS and Kernighan-Lin), storage nodes are divided into different tiers: a high-performance tier for storing high-priority, frequently accessed data blocks; a high-capacity tier for storing compressed, less frequently accessed data blocks; and a standby tier for failover and disaster recovery backup nodes. The principles of this tiered partitioning include fully considering node load balancing, minimizing cross-node transmission path overhead, and optimizing data block storage efficiency.

[0117] Based on the constructed cross-node storage graph model, the graph optimization algorithm is used to generate the optimal block storage distribution solution;

[0118] Specifically, based on the constructed cross-node storage graph model, a graph optimization algorithm (such as the shortest path algorithm Dijkstra or the minimum spanning tree algorithm) is used to generate an optimal data block storage distribution solution.

[0119] High-priority data blocks: have the lowest path weight, the shortest transmission delay, and are preferentially assigned to high-performance nodes;

[0120] Low-priority data blocks: have high path overhead but low storage cost, and are allocated to large-capacity cold storage nodes.

[0121] For detected duplicate data blocks, the system only stores one physical copy; in other storage nodes, the reference relationship of duplicate data blocks is managed through logical pointers (such as soft links or metadata mapping tables) to reduce storage redundancy.

[0122] Through the reinforcement learning algorithm, the node storage status and the distribution status of the access blocks are monitored in real time, and the distribution storage strategy of the deduplicated data blocks is dynamically adjusted.

[0123] Furthermore, the formula of the layered graph partitioning algorithm is as follows:

[0124]

[0125] Where min F represents the objective function of hierarchical graph partitioning, which is used to describe the optimization goal of data block storage distribution; N represents the total number of data blocks; M represents the total number of storage nodes; represents the set of neighboring data blocks of data block i; w ij represents the transmission weight between data block i and data block j; d ij represents the network delay or transmission path weight between different nodes; f ij represents the similarity characteristics of data blocks i and j; c represents the weight coefficient, which is used to balance the optimization priority of transmission cost and node load balancing; L k represents the current load of storage node k; C k Represents the capacity of storage node k.

[0126] S5: Based on the reinforcement learning algorithm, the performance of the deduplication process is monitored in real time, and the block strategy, indexing frequency and storage distribution are dynamically optimized to achieve global optimization of performance and resource utilization. The reinforcement learning algorithm is based on a two-layer hierarchical optimization framework. The top layer uses the policy gradient method to optimize the global performance goals of the deduplication process. The bottom layer combines the multi-agent collaboration mechanism to achieve coordinated dynamic adjustment of the block strategy, indexing frequency and storage mapping.

[0127] Specifically, through reinforcement learning, agents learn how to select optimal actions under different system states to optimize overall performance. A two-layer reinforcement learning framework is designed: top-level optimization is responsible for global policy decisions and performance optimization. Bottom-level optimization is responsible for dynamically adjusting block partitioning, indexing, and storage layout parameters, and enabling multi-agent collaboration.

[0128] Top-level optimization primarily makes decisions based on global objectives. Specific implementation steps include setting overall performance targets based on data deduplication requirements, such as maximizing the deduplication rate, improving storage space utilization, and reducing data processing latency. The top-level reinforcement learning agent regularly analyzes the current system state and dynamically optimizes global parameters, such as the granularity range of the chunking strategy and the index update cycle. For example, when system load is high, it prioritizes reducing chunk size to speed up data processing; when system bandwidth pressure is high, it reduces the frequency of index updates to conserve resources.

[0129] The underlying layer uses multi-agent reinforcement learning, and each agent is responsible for block strategy, index maintenance and storage layout optimization. The specific implementation is as follows: monitor the characteristic distribution of data blocks and the current system performance, and dynamically adjust the granularity of blocks; dynamically adjust the frequency and strategy of index updates according to the frequency of data changes and system resource status; monitor the load of storage nodes and the frequency of data access, and dynamically adjust the storage layout of data blocks; multiple agents share system status information, and through collaborative decision-making, ensure that block strategies, index updates and data storage layouts cooperate with each other to achieve global optimization.

[0130] In summary, this invention effectively identifies data redundancy patterns and compression potential through a dynamic block algorithm and multidimensional feature extraction, and performs deduplication using an edge entropy optimal deduplication algorithm, significantly reducing data redundancy and improving storage space utilization. The dynamic block algorithm, combined with the data content change rate and block granularity adjustment factor, performs adaptive granularity optimization for different data types (such as network data streams and large files), balancing storage efficiency and deduplication accuracy.

[0131] The present invention maintains a global index through a distributed hash table (DHT) and a consensus protocol, achieving efficient mapping between unique identifiers and storage locations for data blocks. A two-layer index structure (primary index and secondary index) is employed to simultaneously support efficient unique data location and similarity retrieval. Deep learning algorithms (such as variational autoencoders, density clustering, and graph neural networks) are used to perform feature dimensionality reduction, similarity clustering, and redundancy analysis on data blocks, effectively improving the accuracy of duplicate data identification, particularly achieving efficient deduplication on a global scale.

[0132] This paper uses a distributed graph neural network and cross-node communication mechanism to achieve distributed collaborative processing of data block deduplication tasks, avoiding centralized bottlenecks and improving system processing efficiency and scalability. It uses a layered graph partitioning algorithm to distribute deduplicated data blocks to different nodes. Combined with a hot and cold tiered storage strategy and node load monitoring, this ensures efficient data storage and access, reduces network transmission overhead, and achieves storage load balancing.

[0133] The present invention uses a reinforcement learning algorithm to monitor the system status in real time and optimize the block strategy, index frequency and storage layout to ensure the global optimization of performance and resource utilization and adapt to changes in system load and data access patterns. Through a consistency protocol (such as the Raft protocol) and an index copy mechanism, it is ensured that when a node fails or a new node is added, the index and data can be correctly synchronized and restored, ensuring the high reliability and disaster recovery capabilities of the system. Using an incremental data flow model, new data blocks are identified in real time, and a compression algorithm is combined to efficiently store non-duplicate data, further reducing storage space occupancy and improving data backup efficiency.

[0134] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A data deduplication method for a cloud backup platform based on distributed storage, characterized in that: The following steps are involved: S1: Adaptively partitions backup data using a dynamic partitioning algorithm, generates a unique identification code for each partition, and extracts multi-dimensional feature information based on an intelligent feature extraction algorithm, including content distribution features, redundancy pattern features, and compression potential assessment features. S2: A global index is established through a distributed hash table, mapping the unique identifier of each block to its storage location, and a distributed consistency protocol is used to dynamically maintain the consistency of the index; S3: Based on the global index, a deep learning algorithm is used to perform redundancy analysis and similarity matching on the block features, identify duplicate or similar data blocks globally, and implement a deduplication strategy based on the detection results; S4: Based on access frequency and load, a layered graph partitioning algorithm is used to map the deduplicated data blocks into a cross-node storage graph model, and the deduplicated data blocks are distributed and stored on multiple nodes. S5: Based on reinforcement learning algorithms, it monitors the performance of the deduplication process in real time and dynamically optimizes the block strategy, indexing frequency, and storage distribution to achieve global optimization of performance and resource utilization.

2. The data deduplication method for a cloud backup platform based on distributed storage according to claim 1, characterized in that: The construction process of the dynamic block algorithm includes the following steps: Based on the content characteristics of the data to be backed up, the sliding window technology is combined with the sensitivity adjustment mechanism to dynamically segment the data and generate an initial block set with boundary adaptability. The initial block set is divided through a multi-level block optimization strategy, combined with the data content change rate and block granularity adjustment factor; Generate a unique identification code for each block, where the identification code is generated based on a cryptographic hash algorithm and a joint calculation of the block characteristics; After the block segmentation is completed, the characteristics of each block are evaluated, and the block segmentation strategy is dynamically adjusted based on the block stability index and specific content distribution rules.

3. The data deduplication method for a cloud backup platform based on distributed storage according to claim 1, characterized in that: The step S3 comprises the following steps: Obtain multi-dimensional feature information of the blocks based on the global index, and generate high-dimensional feature vectors using feature embedding algorithms; Adaptive variational autoencoders are used to reduce the dimensionality of high-dimensional feature vectors to generate a latent feature space representation. Density clustering analysis is used to evaluate the similarity and repetitiveness between blocks and to construct a similarity matrix of block features. Based on the similarity matrix, a distributed graph neural network is used to dynamically construct a block feature graph, perform redundancy detection and duplicate clustering analysis on the blocks, and improve the efficiency and accuracy of duplicate detection through a cross-node distributed collaboration mechanism; According to the duplicate detection results, the edge entropy optimal deduplication algorithm is used to perform deduplication operations on duplicate or similar data blocks, delete completely duplicate blocks, and implement a merge storage strategy for similar blocks.

4. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 3, characterized in that: The formula of the distributed graph neural network is as follows: in, Represents the feature vector of block v after being updated by the graph neural network; and Represents the multidimensional feature data of data blocks; F u and F v represents the additional feature vectors of nodes u and v, including relevant metadata obtained from the global index; N(v) represents the set of neighbor nodes of node v; d u and d v Indicates the number of other similar blocks it is connected to; represents the attention weight, that is, the similarity between node u and node v; W (l) Represents the trainable weight matrix of the graph neural network at layer I; represents the adaptive weight coefficient; σ represents the activation function.

5. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 1, characterized in that: The step S4 comprises the following steps: Based on the deduplicated data blocks, the importance of the data blocks is dynamically evaluated using access frequency and load balancing strategies to calculate the storage priority of each data block. Based on the characteristics of the deduplicated data blocks and the node load status, a cross-node storage graph model is constructed using a layered graph partitioning algorithm. In the graph, nodes represent storage nodes, and edges represent the transmission path weights between nodes. Based on the constructed cross-node storage graph model, the graph optimization algorithm is used to generate the optimal block storage distribution solution; Through the reinforcement learning algorithm, the node storage status and the distribution status of the access blocks are monitored in real time, and the distribution storage strategy of the deduplicated data blocks is dynamically adjusted.

6. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 5, characterized in that: The formula of the layered graph partitioning algorithm is as follows: Where min F represents the objective function of hierarchical graph partitioning, which is used to describe the optimization goal of data block storage distribution; N represents the total number of data blocks; M represents the total number of storage nodes; represents the set of neighboring data blocks of data block i; w ij represents the transmission weight between data block i and data block j; d ij represents the network delay or transmission path weight between different nodes; f ij represents the similarity characteristics of data blocks i and j; c represents the weight coefficient; L k represents the current load of storage node k; C k Represents the capacity of storage node k.

7. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 1, characterized in that: The step S4 also includes storing the data blocks determined to be duplicates only once, and managing other duplicate references through logical pointers; for newly added non-duplicate data blocks, efficient storage is achieved by combining compression algorithms and hot and cold data tiering strategies, while introducing an incremental data flow model to dynamically optimize the storage layout.

8. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 1, characterized in that: The establishment of the global index supports efficient associative mapping between high-dimensional features and physical storage locations through a multi-dimensional index mapping mechanism combined with a content-aware hash algorithm based on block features.

9. The method for deduplicating data on a cloud backup platform based on distributed storage according to claim 1, characterized in that: The reinforcement learning algorithm is based on a two-layer hierarchical optimization framework. The top layer uses a policy gradient method to optimize the global performance objectives of the deduplication process, and the bottom layer combines a multi-agent collaboration mechanism to achieve coordinated dynamic adjustment of the blocking strategy, indexing frequency and storage mapping.

Images