A learning-based index-based data deduplication method and system

Abstract

The application discloses a data deduplication method and system based on a learning index, and belongs to the technical field of computer storage. The application uses a learning index structure to replace a traditional hash table. The learning index structure is a tree structure, and comprises model nodes and data nodes. An ordered fingerprint array and a buffer are arranged in each data node. The application arranges the fingerprints compactly and uses a buffer with a fixed size, thereby reducing the memory overhead of the index. When the number of fingerprints stored in the buffer of any leaf node reaches a preset number, a retraining mechanism is triggered, the model parameters and the structure layout are continuously optimized, and the index can maintain high-precision prediction and stable performance in a long-term running process. In addition, the application adopts a lightweight asynchronous retraining mechanism, so that the foreground can continue to perform query when the model is retrained, and the performance bottleneck of the traditional hash table in the expansion and rehashing process is avoided.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, and more specifically, relates to a data deduplication method and system based on a learning index. Background Technology

[0002] With the exponential growth of global data volume, managing and storing this data in a cost-effective manner has become one of the core challenges of modern storage systems. The rapid growth of global data volume and the arrival of the data deluge require storage systems not only to have the ability to process large-scale data, but also to optimize storage space through efficient storage technologies, especially reducing redundant data, in order to improve storage efficiency.

[0003] Data deduplication technology is a highly efficient compression technique developed to address this problem. Its core objective is to reduce storage requirements by identifying and eliminating redundant data, thereby saving valuable storage space. Especially when storing backup data, data deduplication can significantly reduce storage needs and improve processing efficiency, making it an indispensable technology in large-scale storage systems. Therefore, researching a data deduplication method is of great significance.

[0004] Typical deduplication systems divide the input data stream (such as backup files, database snapshots, virtual machine images, etc.) into multiple data chunks and uniquely identify and deduplicate each chunk using cryptographic hash signatures (such as SHA-256). These hash signatures, also known as fingerprints, enable deduplication techniques to eliminate redundant data at the block level (e.g., 8KB) or the file level. However, in existing data deduplication methods, hash tables, as the key data structure for storing data block fingerprints (hash values), face some significant shortcomings, especially in terms of memory consumption. As the amount of data increases, the memory overhead of hash tables rises sharply. Therefore, optimizing the memory usage of hash tables has become a major challenge for data deduplication technology. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a data deduplication method and system based on learning index to solve the technical problem of high memory consumption in the existing technology.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a data deduplication method based on a learned index, comprising:

[0007] A1. Divide the input data stream into multiple data blocks, calculate the cryptographic hash of each data block, and use it as the corresponding fingerprint;

[0008] A2. For each data block, check if its fingerprint exists in the learning index structure. If it exists, the data block is determined to be duplicate data and its contents are not stored. Otherwise, the fingerprint of the data block is inserted into the index structure and the data block is stored.

[0009] The learning index structure is a tree structure, where the leaf nodes are data nodes, including: a first linear model, a buffer of a preset size, and an ordered data array for storing data block fingerprints; the other nodes in the index structure besides the leaf nodes are model nodes, including: a second linear model and a pointer array for storing pointers to its child nodes; the first linear model is used to map the input fingerprint to the predicted position of the fingerprint in the corresponding ordered data array; the second linear model is used to map the input fingerprint to the position of the pointer of the target child node in the corresponding pointer array; the target child node is the ancestor node of the leaf node storing the corresponding input fingerprint;

[0010] Methods for checking if any fingerprint fp exists in an index structure include:

[0011] The search for fingerprint fp begins from the root node of the index structure and continues until a leaf node is reached. This yields the predicted fingerprint fp' at the ordered data array A of the corresponding leaf node. If fp and fp' are the same, the fingerprint fp is determined to exist in the index structure. If fp and fp' are different, the search for fp in array A is performed. If fp exists, the fingerprint fp is determined to exist in the index structure. If fp does not exist, the search for fp in the buffer B of the corresponding leaf node is performed. If fp exists in buffer B, the fingerprint fp is determined to exist in the index structure. If fp does not exist in buffer B, the fingerprint fp is determined not to exist in the index structure.

[0012] The method for inserting any fingerprint fp into the index structure includes: obtaining the leaf node where the fingerprint fp is located, and inserting the fingerprint fp into the buffer of that leaf node;

[0013] The above data deduplication method also includes: when any leaf node When the number of fingerprints stored in the buffer reaches a preset number, the leaf nodes... Retrain the first linear model: Fingerprints in the buffer are merged into In the ordered data array, and update The first linear model.

[0014] More preferably, for leaf nodes The retraining operation for the first linear model includes:

[0015] Create a new array A', and The fingerprints in the buffer and the ordered data array are placed sequentially in array A';

[0016] By fitting the fingerprints in array A' to their positions in array A', the corresponding fitting model is obtained. ; Calculate the fitting model The fitting error err;

[0017] When the fitting error err is less than or equal to the preset error, a new node is constructed. , fit the model As its first linear model, array A' is used as its ordered data array, and the index structure is... The parent node points to ;

[0018] When the fitting error err is greater than the preset error, and The parent node has only one pointer pointing to At that time, a reconstruction operation is performed on array A':

[0019] Array A' is split multiple times, each time dividing the current array into two arrays to form a split tree. For each array in the split tree, the fingerprint in the array is fitted to its position within the array to obtain a corresponding fitting model. The fitting error of the fitting model is weighted and summed with the space occupied when constructing leaf nodes using the array, and this sum is taken as the cost of the array. The layer with the minimum total cost among all arrays in the split tree is selected as the candidate layer. The subtree from the first layer to the candidate layer in the split tree is obtained, and the subtree is pruned to minimize the total cost of all leaf nodes in the subtree while containing all fingerprints in array A'. For each leaf node in the pruned subtree... The array below Determine whether the fitting error of the corresponding fitting model is less than or equal to the preset error. If so, then move the leaf node... Mark key nodes as arrays Build new nodes The fitted model corresponding to this array is taken as its first linear model, and this array is taken as its ordered data array; otherwise, the array is used for further processing. Perform a refactoring operation;

[0020] Record all nodes in the candidate layers obtained during the entire process of reconstructing array A', forming a candidate node set S; construct a new model node. and make the index structure The parent node points to the model node. ; build pointer array Its size is the number of nodes n in set S; All pointers stored in the array correspond one-to-one with all candidate nodes in set S; for Each pointer stored in the middle, when its corresponding candidate node When it is a critical node, point the pointer to the node. The corresponding new node When its corresponding candidate node If it is not a critical node, point the pointer to the node. Nodes in the split tree The marker is the new node corresponding to the ancestor node of the key node. ; It also includes a second linear model for mapping the input fingerprint to the position of the pointer to the target child node in the corresponding pointer array;

[0021] When the fitting error err is greater than the preset error, and The parent node has multiple pointers pointing to Then, the array A' is divided into two arrays, and for each array, the relationship between the fingerprints in that array and their positions within that array is fitted to obtain the corresponding fitting model; a new node is constructed for each array. The fitted model corresponding to this array is taken as its first linear model, this array is taken as its ordered data array, and the index structure is made... The parent node points to .

[0022] More preferably, for any array r, the method of dividing it into two arrays includes:

[0023] Construct a method to map each fingerprint in array r to Linear functions within the range ; where y ; b are both preset coefficients; x represents the fingerprint;

[0024] Substitute y=1 into In the above, x is solved and used as the dividing point of array r, thus dividing array r into two arrays;

[0025] at this time, The second linear model in is This is used to map the input fingerprint x to the position of the pointer to the target child node in the corresponding pointer array. .

[0026] More preferably, the retraining operation is performed by creating a new thread, and the leaf nodes are retrained before the retraining operation. The upper read-only lock is used on the leaf nodes after the retraining operation is completed. Unlock.

[0027] More preferably, at the beginning of each retraining operation, it is determined whether there is a query operation on the leaf node that was previously retrained. If there is no query operation, it is recycled; otherwise, it is not recycled.

[0028] More preferably, the buffer is a linear hash.

[0029] More preferably, in the above A2, the exponential search method is used to search for whether fp exists in array A.

[0030] Secondly, the present invention provides a data deduplication system based on a learning index, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the data deduplication method provided in the first aspect of the present invention when executing the computer program.

[0031] Thirdly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device containing the storage medium to perform the data deduplication method provided in the first aspect of the present invention.

[0032] Fourthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the data deduplication method provided in the first aspect of the invention.

[0033] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0034] 1. This invention provides a data deduplication method based on a learning index. It replaces the traditional hash table with a learning index structure, introducing a learnable linear model into fingerprint index management to achieve predictive lookup of fingerprint locations. The learning index structure is a tree structure, including model nodes and data nodes. Each data node contains an ordered fingerprint array and a buffer. This invention reduces the memory overhead of the index by compactly arranging fingerprints and using a fixed-size buffer. Furthermore, when the number of fingerprints stored in the buffer of any leaf node reaches a preset number, a retraining mechanism is triggered to continuously optimize model parameters and structural layout, ensuring that the index maintains high-precision prediction and stable performance during long-term operation.

[0035] 2. Furthermore, the data deduplication method provided by this invention introduces an adaptive splitting decision for node range during the retraining process. During the splitting process, the weighted sum of the fitting error of the array's fitting model and the space occupied when constructing leaf nodes using the corresponding array is used as the overhead of the corresponding array. This dynamically balances the search cost and space overhead of the index, enabling timely merging of data in the buffer into the ordered data array. It can also adaptively update for different loads to obtain better model parameters and structural layout, thus better ensuring that the index maintains high-precision prediction and stable performance during long-term operation.

[0036] 3. Furthermore, considering that in deduplication scenarios, two consecutive insertions are almost never inserted into the same leaf node, the data deduplication method provided by this invention adopts a lightweight asynchronous retraining mechanism. The retraining operation is performed by creating a new thread; and a read-only lock is placed on the leaf node before the retraining operation, and the leaf node is unlocked after the retraining operation is completed, so that the foreground can continue to query during model retraining, avoiding the performance bottleneck of traditional hash tables in the process of expansion and rehashing.

[0037] 4. Furthermore, the data deduplication method provided by this invention uses a linear hash table as a buffer, which has better search and insertion performance while ensuring low memory overhead.

[0038] 5. Furthermore, the data deduplication method provided by the present invention, in A2 above, uses the exponential search method to search for whether fp exists in array A, which is more efficient and significantly improves the search performance. Attached Figure Description

[0039] Figure 1 A schematic diagram illustrating the framework of the data deduplication method based on a learning index provided in an embodiment of the present invention;

[0040] Figure 2 This is a flowchart illustrating the data deduplication method based on a learning index provided in an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of asynchronous retraining provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] To achieve the above objectives, in a first aspect, the present invention provides a data deduplication method based on a learned index, comprising:

[0044] A1. Divide the input data stream into multiple data blocks, calculate the cryptographic hash of each data block, and use it as the corresponding fingerprint to uniquely identify the content of the data block.

[0045] It should be noted that each data block can be a fixed size or a variable-size block divided based on content boundaries; this is not limited here. There are various ways to calculate the cryptographic hash of a data block, such as SHA-256 and SHA-1; this is not limited here either.

[0046] A2. For each data block, check if its fingerprint exists in the learning index structure. If it exists, the data block is determined to be duplicate data and its contents are not stored. Otherwise, the fingerprint of the data block is inserted into the index structure and the data block is stored.

[0047] The learning index structure is a tree structure, with leaf nodes being data nodes, including: a first linear model, a buffer of a preset size, and an ordered data array for storing data block fingerprints; other nodes in the index structure besides the leaf nodes are model nodes, including: a second linear model and a pointer array for storing pointers to its child nodes; the first linear model is used to map the input fingerprint to the predicted position of the fingerprint in the corresponding ordered data array; the second linear model is used to map the input fingerprint to the position of the pointer of the target child node in the corresponding pointer array; the target child node is the ancestor node of the leaf node storing the corresponding input fingerprint.

[0048] Preferably, in one optional implementation, the buffer is a linear hash, which provides better search and insertion performance while ensuring low memory overhead.

[0049] In step A2, the method for checking whether any fingerprint fp exists in the index structure includes:

[0050] The search for fingerprint fp begins from the root node of the index structure and continues until a leaf node is reached. This yields the predicted fingerprint fp' at the ordered data array A of the corresponding leaf node. If fp and fp' are the same, the fingerprint fp is determined to exist in the index structure. If fp and fp' are different, the search for fp in array A is performed. If fp exists, the fingerprint fp is determined to exist in the index structure. If fp does not exist, the search for fp in the buffer B of the corresponding leaf node is performed. If fp exists in buffer B, the fingerprint fp is determined to exist in the index structure. If fp does not exist in buffer B, the fingerprint fp is determined not to exist in the index structure.

[0051] It should be noted that there are multiple ways to check if fp exists in array A. In one optional implementation method, array A is directly traversed to search for fp. In another optional implementation method, the ordered nature of array A can be used to determine the size relationship between fp and fp'. When fp is less than fp', the search in array A starts from fp' and proceeds in smaller directions. If a fingerprint smaller than fp is found but fp is not found, then the fingerprint fp is determined not to exist in array A; otherwise, the fingerprint fp is determined to exist. When fp is greater than fp', the search in array A starts from fp' and proceeds in larger directions. If a fingerprint greater than fp is found but fp is not found, then the fingerprint fp is determined not to exist in array A; otherwise, the fingerprint fp is determined to exist. In a third optional implementation method, an exponential search method is used to check if fp exists in array A. The third optional implementation method is preferred as it is more efficient and significantly improves search performance.

[0052] The method for inserting any fingerprint fp into the index structure includes: obtaining the leaf node where the fingerprint fp is located, and inserting the fingerprint fp into the buffer of that leaf node.

[0053] The above data deduplication method also includes: when any leaf node When the number of fingerprints stored in the buffer reaches a preset number, the leaf nodes... Retrain the first linear model: Fingerprints in the buffer are merged into In the ordered data array, and update The first linear model.

[0054] In one alternative implementation, the leaf nodes The retraining operation for the first linear model includes:

[0055] Create a new array A', and The fingerprints in the buffer and the ordered data array are placed sequentially in array A';

[0056] By fitting the fingerprints in array A' to their positions in array A', the corresponding fitting model is obtained. ; Calculate the fitting model The fitting error err;

[0057] When the fitting error err is less than or equal to the preset error, a new node is constructed. , fit the model As its first linear model, array A' is used as its ordered data array, and the index structure is... The parent node points to ;

[0058] When the fitting error err is greater than the preset error, and The parent node has only one pointer pointing to At that time, a reconstruction operation is performed on array A':

[0059] Array A' is split multiple times, each time dividing the current array into two arrays to form a split tree. For each array in the split tree, the fingerprint in the array is fitted to its position within the array to obtain a corresponding fitting model. The fitting error of the fitting model is weighted and summed with the space occupied when constructing leaf nodes using the array, and this sum is taken as the cost of the array. The layer with the minimum total cost among all arrays in the split tree is selected as the candidate layer. The subtree from the first layer to the candidate layer in the split tree is obtained, and the subtree is pruned to minimize the total cost of all leaf nodes in the subtree while containing all fingerprints in array A'. For each leaf node in the pruned subtree... The array below Determine whether the fitting error of the corresponding fitting model is less than or equal to the preset error. If so, then move the leaf node... Mark key nodes as arrays Build new nodes The fitted model corresponding to this array is taken as its first linear model, and this array is taken as its ordered data array; otherwise, the array is used for further processing. Perform a refactoring operation;

[0060] Record all nodes in the candidate layers obtained during the entire process of reconstructing array A', forming a candidate node set S; construct a new model node. and make the index structure The parent node points to the model node. ; build pointer array Its size is the number of nodes n in set S; All pointers stored in the array correspond one-to-one with all candidate nodes in set S; for Each pointer stored in the middle, when its corresponding candidate node When it is a critical node, point the pointer to the node. The corresponding new node When its corresponding candidate node If it is not a critical node, point the pointer to the node. Nodes in the split tree The marker is the new node corresponding to the ancestor node of the key node. ; It also includes a second linear model for mapping the input fingerprint to the position of the pointer to the target child node in the corresponding pointer array;

[0061] When the fitting error err is greater than the preset error, and The parent node has multiple pointers pointing to Then, the array A' is divided into two arrays, and for each array, the relationship between the fingerprints in that array and their positions within that array is fitted to obtain the corresponding fitting model; a new node is constructed for each array. The fitted model corresponding to this array is taken as its first linear model, this array is taken as its ordered data array, and the index structure is made... The parent node points to .

[0062] By using the above method, the data in the buffer can be merged into the ordered data array in a timely manner, and adaptive updates can be performed for different loads.

[0063] It should be noted that the above-mentioned fitting methods can be various, such as least squares method, ridge regression method, Lasso regression method, etc., and are not limited here, but least squares method is preferred. There are various ways to measure the fitting error of the above-mentioned fitting model, such as mean square error, mean absolute error, root mean square error, etc., and are not limited here. Preferably, in one optional implementation, the fitting error of the fitting model corresponding to any array r is: ; For the first element in array r i The error loss value between the predicted position and the actual position of each fingerprint in the corresponding ordered data array; The number of fingerprints in array r.

[0064] In one alternative implementation, the retraining operation is performed by creating a new thread; and a read-only lock is placed on the leaf nodes before the retraining operation, and on the leaf nodes after the retraining operation is completed. Unlocking is performed to enable the front-end to continue querying during model retraining through a lightweight asynchronous retraining mechanism, thus avoiding the performance bottleneck of traditional hash tables during expansion and rehashing.

[0065] It should be noted that there are multiple ways to divide any array r into two arrays, and no limitation is made here. Preferably, in one optional implementation, the methods for dividing any array r into two arrays include:

[0066] Construct a method to map each fingerprint in array r to Linear functions within the range ; where y ; b are both preset coefficients; x represents the fingerprint;

[0067] Substitute y=1 into In the above, x is solved and used as the dividing point of array r, thus dividing array r into two arrays;

[0068] at this time, The second linear model in is This is used to map the input fingerprint x to the position of the pointer to the target child node in the corresponding pointer array. .

[0069] In one alternative implementation, at the start of each retraining operation, the leaf nodes that previously underwent retraining operations are identified. If there is no query operation, the data will be recycled; otherwise, it will not be recycled.

[0070] In summary, this invention uses a learned index to replace the hash table in traditional deduplication methods, achieving higher performance and lower memory consumption. The learned index employs a multi-level tree structure, including model nodes and data nodes. Each data node contains an ordered fingerprint array and a buffer for quickly determining the existence of fingerprints and temporarily storing newly added fingerprint data. The ordered fingerprint array is compactly arranged, maintaining efficient search and insertion performance while consuming little memory. A retraining mechanism triggered under specific conditions continuously optimizes model parameters and structural layout, ensuring that the index structure maintains high-accuracy prediction and stable performance over long-term operation.

[0071] To further illustrate the data deduplication method provided by the present invention, a specific embodiment is described in detail below:

[0072] like Figure 1 and Figure 2 As shown, the data deduplication method provided in this embodiment includes the following steps:

[0073] A1. The input data stream is divided into multiple data blocks, and a cryptographic hash is calculated for each data block as its corresponding fingerprint. To avoid the boundary shifting problem common in fixed-size blocks, this embodiment uses the FastCDC content-defined chunking algorithm to adaptively determine block boundaries based on the data content, thereby ensuring that similar data maintains high block boundary consistency across different versions. The system sets the average block size to 4KB and uses a minimum block size of 512B and a maximum block size of 64KB constraint mechanism to avoid excessive splitting or merging. After block division, a cryptographic hash fingerprint is calculated for each data block. This embodiment uses the SHA-256 algorithm to generate a 256-bit fingerprint value to uniquely identify the data block content.

[0074] A2. For each data block, check if its fingerprint exists in the learning index structure. If it exists, the data block is determined to be duplicate data and its contents are not stored. Otherwise, the fingerprint of the data block is inserted into the index structure and the data block is stored.

[0075] The learning index structure is a tree structure, where the leaf nodes are data nodes, including: a first linear model, a buffer of a preset size, and an ordered data array for storing data block fingerprints; the other nodes in the index structure besides the leaf nodes are model nodes, including: a second linear model and a pointer array for storing pointers to its child nodes; the first linear model is used to map the input fingerprint to the predicted position of the fingerprint in the corresponding ordered data array; the second linear model is used to map the input fingerprint to the position of the pointer of the target child node in the corresponding pointer array; the target child node is the ancestor node of the leaf node storing the corresponding input fingerprint; in this embodiment, the format of the first linear model and the second linear model is... .

[0076] In this embodiment, when initializing the index structure, if deduplication is to be performed based on the previous deduplication results, the previous fingerprints are formed into an ordered array, and the initial index structure is constructed according to the reconstruction operation in the retraining operation; otherwise, an empty index structure is directly constructed.

[0077] In this embodiment, the method for checking whether any fingerprint fp exists in the index structure includes:

[0078] The search for fingerprint fp begins from the root node of the index structure and continues until a leaf node is reached. This yields the predicted fingerprint fp' at the ordered data array A of the corresponding leaf node. If fp and fp' are the same, the fingerprint fp is determined to exist in the index structure. If fp and fp' are different, the search for fp in array A is performed. If fp exists, the fingerprint fp is determined to exist in the index structure. If fp does not exist, the search for fp in the buffer B of the corresponding leaf node is performed. If fp exists in buffer B, the fingerprint fp is determined to exist in the index structure. If fp does not exist in buffer B, the fingerprint fp is determined not to exist in the index structure.

[0079] In this embodiment, the exponential search method is used to search for whether fp exists in array A.

[0080] In this embodiment, the method for inserting any fingerprint fp into the index structure includes: obtaining the leaf node where the fingerprint fp is located, and inserting the fingerprint fp into the buffer of the leaf node;

[0081] The above data deduplication method also includes: when any leaf node When the number of fingerprints stored in the buffer reaches a preset number, the leaf nodes... Retrain the first linear model: Fingerprints in the buffer are merged into In the ordered data array, and update The first linear model. In this embodiment, the preset quantity is 0.9 of the buffer capacity.

[0082] Specifically, in this embodiment, for leaf nodes The retraining operation for the first linear model includes:

[0083] Create a new array A', and The fingerprints in the buffer and the ordered data array are placed sequentially in array A';

[0084] By fitting the fingerprints in array A' to their positions in array A', the corresponding fitting model is obtained. ; Calculate the fitting model The fitting error err;

[0085] 1) When the fitting error err is less than or equal to the preset error, construct a new node. , fit the model As its first linear model, array A' is used as its ordered data array, and the index structure is... The parent node points to ;

[0086] 2) When the fitting error err is greater than the preset error, and The parent node has only one pointer pointing to At that time, a reconstruction operation is performed on array A':

[0087] Array A' is split multiple times, each time dividing the current array into two arrays to form a split tree. For each array in the split tree, the fingerprint in the array is fitted to its position within the array to obtain a corresponding fitting model. The fitting error of the fitting model is weighted and summed with the space occupied when constructing leaf nodes using the array, and this sum is taken as the cost of the array. The layer with the minimum total cost among all arrays in the split tree is selected as the candidate layer. The subtree from the first layer to the candidate layer in the split tree is obtained, and the subtree is pruned to minimize the total cost of all leaf nodes in the subtree while containing all fingerprints in array A'. For each leaf node in the pruned subtree... The array below Determine whether the fitting error of the corresponding fitting model is less than or equal to the preset error. If so, then move the leaf node... Mark key nodes as arrays Build new nodes The fitted model corresponding to this array is taken as its first linear model, and this array is taken as its ordered data array; otherwise, the array is used for further processing. Perform a refactoring operation;

[0088] Record all nodes in the candidate layers obtained during the entire process of reconstructing array A', forming a candidate node set S; construct a new model node. and make the index structure The parent node points to the model node. ; build pointer array Its size is the number of nodes n in set S; All pointers stored in the array correspond one-to-one with all candidate nodes in set S; for Each pointer stored in the middle, when its corresponding candidate node When it is a critical node, point the pointer to the node. The corresponding new node When its corresponding candidate node If it is not a critical node, point the pointer to the node. Nodes in the split tree The marker is the new node corresponding to the ancestor node of the key node. ; It also includes a second linear model for mapping the input fingerprint to the position of the pointer to the target child node in the corresponding pointer array;

[0089] 3) When the fitting error err is greater than the preset error, and The parent node has multiple pointers pointing to Then, the array A' is divided into two arrays, and for each array, the relationship between the fingerprints in that array and their positions within that array is fitted to obtain the corresponding fitting model; a new node is constructed for each array. The fitted model corresponding to this array is taken as its first linear model, this array is taken as its ordered data array, and the index structure is made... The parent node points to .

[0090] In this embodiment, the method for dividing any array r into two arrays includes:

[0091] Construct a method to map each fingerprint in array r to Linear functions within the range ; where y ; b are both preset coefficients; x represents the fingerprint;

[0092] Substitute y=1 into In the above, x is solved and used as the dividing point of array r, thus dividing array r into two arrays;

[0093] at this time, The second linear model in is This is used to map the input fingerprint x to the position of the pointer to the target child node in the corresponding pointer array. .

[0094] In this embodiment, the fitting method used is the least squares method; the formula for calculating the fitting error of the fitting model corresponding to any array r is: ; For the first element in array r i The error loss value between the predicted position and the actual position of each fingerprint in the corresponding ordered data array; Let r be the number of fingerprints in array r. Additionally, to reflect the different importance of fitting error and leaf node memory consumption, both are assigned weights. In this embodiment, the weight of fitting error is 20, the weight of memory consumption is 2, and the preset error is 5.

[0095] In this embodiment, the retraining operation is performed by creating a new thread; and the leaf nodes are processed before the retraining operation. The upper read-only lock is used on the leaf nodes after the retraining operation is completed. Unlock.

[0096] The specific implementation is as follows: At the start of the retraining operation, an atomic variable is used to set the leaf node to the "retraining in progress" state. Other threads accessing the node check this state before the insertion operation. If retraining is detected, they enter a waiting state using a mutex lock and a condition variable, thus achieving read-only locking of the leaf node. After retraining is completed, the state is reset and broadcast to wake up all waiting threads, allowing them to continue accessing the node, which is equivalent to unlocking the leaf node.

[0097] like Figure 3 As shown, an insertion causes the buffer to reach a threshold, triggering a background thread to perform a retraining operation. During the retraining process, the lookup for this node is still performed on the old node, but the insertion for this node will be blocked. After the retraining is completed, the new node replaces the old node and wakes up the blocked thread.

[0098] In this embodiment, at the beginning of each retraining operation, the leaf nodes that have previously undergone retraining operations are identified. If a query operation exists on a leaf node, it is recycled; otherwise, it is not recycled. In this embodiment, since the search is single-threaded, a boolean variable is used to record whether a query operation exists on a leaf node.

[0099] In this embodiment, the buffer is a linear hash. When constructing a new node, if the number of fingerprints of the node is less than 16384, the preset size of the buffer is 2048 fingerprints; otherwise, the preset size of the buffer is 4096 fingerprints.

[0100] In this embodiment, to facilitate data recovery, in step A2 above, a fingerprint of each data block is also stored. To recover the source file, the specific physical data blocks are located based on the fingerprint composition of the source file, and finally, the data blocks are assembled to recover the source file.

[0101] In summary, to achieve better performance, this embodiment introduces a cost model to calculate overhead in the design of the learning index, dynamically determining the data partitioning method and node type to achieve an optimal balance between search cost and memory overhead. Furthermore, by employing exponential search when searching the ordered fingerprint array, search performance is significantly improved; a linear hash table is used as a buffer, ensuring excellent search and insertion performance while maintaining low memory overhead; and a retraining mechanism is triggered under specific conditions to continuously optimize model parameters and structural layout, ensuring that the index maintains high-precision prediction and stable performance during long-term operation.

[0102] Secondly, the present invention provides a data deduplication system based on a learning index, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the data deduplication method provided in the first aspect of the present invention when executing the computer program.

[0103] The related technical solutions are the same as the data deduplication method provided in the first aspect of this invention, and will not be described in detail here.

[0104] Thirdly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device containing the storage medium to perform the data deduplication method provided in the first aspect of the present invention.

[0105] The related technical solutions are the same as the data deduplication method provided in the first aspect of this invention, and will not be described in detail here.

[0106] Fourthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the data deduplication method provided in the first aspect of the invention.

[0107] The related technical solutions are the same as the data deduplication method provided in the first aspect of this invention, and will not be described in detail here.

[0108] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data deduplication method based on a learned index, characterized in that, include: A1. Divide the input data stream into multiple data blocks, calculate the cryptographic hash of each data block, and use it as the corresponding fingerprint; A2. For each data block, check if its fingerprint exists in the learning index structure. If it exists, the data block is determined to be duplicate data and its contents are no longer stored. Otherwise, insert the fingerprint of the data block into the index structure and store the data block; The learning index structure is a tree structure, where the leaf nodes are data nodes, including: a first linear model, a buffer of a preset size, and an ordered data array for storing data block fingerprints; the other nodes in the index structure besides the leaf nodes are model nodes, including: a second linear model and a pointer array for storing pointers to its child nodes; the first linear model is used to map the input fingerprint to the predicted position of the fingerprint in the corresponding ordered data array; the second linear model is used to map the input fingerprint to the position of the pointer of the target child node in the corresponding pointer array; the target child node is the ancestor node of the leaf node storing the corresponding input fingerprint; Methods for checking if any fingerprint fp exists in an index structure include: The search for fingerprint fp begins from the root node of the index structure and continues until a leaf node is reached. This yields the predicted fingerprint fp' at the ordered data array A of the corresponding leaf node. If fp and fp' are the same, the fingerprint fp is determined to exist in the index structure. If fp and fp' are different, the search for fp in array A is performed. If fp exists, the fingerprint fp is determined to exist in the index structure. If fp does not exist, the search for fp in the buffer B of the corresponding leaf node is performed. If fp exists in buffer B, the fingerprint fp is determined to exist in the index structure. If fp does not exist in buffer B, the fingerprint fp is determined not to exist in the index structure. The method for inserting any fingerprint fp into the index structure includes: obtaining the leaf node where the fingerprint fp is located, and inserting the fingerprint fp into the buffer of that leaf node; The data deduplication method further includes: when any leaf node When the number of fingerprints stored in the buffer reaches a preset number, the leaf nodes... Retrain the first linear model: Fingerprints in the buffer are merged into In the ordered data array, and update The first linear model.

2. The data deduplication method according to claim 1, characterized in that, leaf nodes The retraining operation for the first linear model includes: Create a new array A', and The fingerprints in the buffer and the ordered data array are placed sequentially in array A'; By fitting the fingerprints in array A' to their positions in array A', the corresponding fitting model is obtained. ; Calculate the fitting model The fitting error err; When the fitting error err is less than or equal to the preset error, a new node is constructed. , fit the model As its first linear model, array A' is used as its ordered data array, and the index structure is... The parent node points to ; When the fitting error err is greater than the preset error, and The parent node has only one pointer pointing to At that time, a reconstruction operation is performed on array A': Array A' is split multiple times, each time dividing the current array into two arrays to form a split tree. For each array in the split tree, the fingerprint in the array is fitted to its position within the array to obtain a corresponding fitting model. The fitting error of the fitting model is weighted and summed with the space occupied when constructing leaf nodes using the array, and this sum is taken as the cost of the array. The layer with the minimum total cost among all arrays in the split tree is selected as the candidate layer. The subtree from the first layer to the candidate layer in the split tree is obtained, and the subtree is pruned to minimize the total cost of all leaf nodes in the subtree while containing all fingerprints in array A'. For each leaf node in the pruned subtree... The array below Determine whether the fitting error of the corresponding fitting model is less than or equal to the preset error. If so, then move the leaf node... Mark key nodes as arrays Build new nodes The fitted model corresponding to this array is taken as its first linear model, and this array is taken as its ordered data array; otherwise, the array is used for further processing. Perform a refactoring operation; Record all nodes in the candidate layers obtained during the entire process of reconstructing array A', forming a candidate node set S; construct a new model node. and make the index structure The parent node points to the model node. ; build pointer array Its size is the number of nodes n in set S; All pointers stored in the array correspond one-to-one with all candidate nodes in set S; for Each pointer stored in the middle, when its corresponding candidate node When it is a critical node, point the pointer to the node. The corresponding new node When its corresponding candidate node If it is not a critical node, point the pointer to the node. Nodes in the split tree The marker is the new node corresponding to the ancestor node of the key node. ; It also includes a second linear model for mapping the input fingerprint to the position of the pointer to the target child node in the corresponding pointer array; When the fitting error err is greater than the preset error, and The parent node has multiple pointers pointing to Then, the array A' is divided into two arrays, and for each array, the relationship between the fingerprints in that array and their positions within that array is fitted to obtain the corresponding fitting model; a new node is constructed for each array. The fitted model corresponding to this array is taken as its first linear model, this array is taken as its ordered data array, and the index structure is made... The parent node points to .

3. The data deduplication method according to claim 2, characterized in that, For any array r, methods to split it into two arrays include: Construct a method to map each fingerprint in array r to Linear functions within the range ; where y ; b are both preset coefficients; x represents the fingerprint; Substitute y=1 into In the above, x is solved and used as the dividing point of array r, thus dividing array r into two arrays; at this time, The second linear model in is This is used to map the input fingerprint x to the position of the pointer to the target child node in the corresponding pointer array. .

4. The data deduplication method according to claim 1, characterized in that, The buffer is a linear hash.

5. The data deduplication method according to claim 1, characterized in that, In A2, the exponential search method is used to search for whether fp exists in array A.

6. The data deduplication method according to any one of claims 1-5, characterized in that, The retraining operation is performed by creating a new thread, and the leaf nodes are retrained before the retraining operation. The upper read-only lock is used on the leaf nodes after the retraining operation is completed. Unlock.

7. The data deduplication method according to any one of claims 1-5, characterized in that, At the start of each retraining operation, it is determined whether there is a query operation on the leaf node that was previously retrained. If there is no query operation, it is recycled; otherwise, it is not recycled.

8. A data deduplication system based on a learning index, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the data deduplication method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to perform the data deduplication method according to any one of claims 1-7.

10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the data deduplication method according to any one of claims 1-7.

Images