Fixed-length data deduplication method, device, equipment and program product

Abstract

The invention relates to a fixed-length data deduplication method and device, computer equipment and a computer program product. The method comprises the following steps: before each piece of fixed-length data is written, segmenting each piece of fixed-length data into a plurality of sub-data blocks; selecting a plurality of sampling data blocks from the plurality of sub-data blocks according to a preset sampling mode; determining a sub-fingerprint corresponding to each sampling data block based on a preset Hash algorithm; the calculation overhead of the preset Hash algorithm is lower than that of the strong Hash algorithm; combining all the sub-fingerprints according to a preset combination mode to obtain similar fingerprints of the fixed-length data; and writing the fixed-length data and the similar fingerprints thereof into a storage medium, and in a background deduplication stage, performing data deduplication based on the similarity of the similar fingerprints of the fixed-length data to be deduplication. According to the method, a background deduplication scheme is improved, the CPU calculation overhead on a data writing path is reduced, I / O performance waste caused by reading non-duplicated data is avoided, the deduplication effect is guaranteed, and meanwhile the influence on the read-write performance of the storage system is minimized.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a fixed-length data deduplication method, a fixed-length data deduplication device, a computer device, and a computer program product. Background Technology

[0002] In data storage, data deduplication is a commonly used data reduction technique that helps storage systems reduce the impact of redundant data on storage costs.

[0003] Based on when the deduplication operation occurs, it can be divided into online deduplication and background deduplication. Background deduplication refers to the process where, when two identical copies of data are written to the storage system, the data is not immediately deleted. Instead, the system periodically performs duplicate data identification and deletion operations in the background after the data has been written to disk. For details, please refer to [link to relevant documentation]. Figure 1 Two Data1 values ​​represent identical data to be deleted, LBA1 and LBA2 represent two different logical addresses, FP represents the fingerprint, and Data... g This represents garbage data; Hash represents the hash algorithm used to calculate the fingerprint. Figure 1 It is known that a traditional background deduplication scheme is as follows: first, the two data are read from the storage medium, and then the fingerprints of each data are calculated. If the fingerprints are the same, the duplicate data is deleted. At this time, the corresponding data in the disk will be marked as garbage data.

[0004] While this solution achieves storage space savings, its inherent drawback is that the background deduplication process requires reading all the original data to be checked from the disk to calculate fingerprints. If the data read is not duplicate data, then this disk I / O operation and its associated computational overhead are pure performance losses, severely consuming the disk bandwidth and CPU resources that should be used to process front-end business I / O, thus having a significant negative impact on the overall performance of the storage system.

[0005] Another backend deduplication solution to address the above problem is to move the fingerprint calculation process forward to before the data is written to disk, such as... Figure 2 As shown in the diagram, in this scheme, a fingerprint is calculated before data is written, and then the data and fingerprint are written to the disk together. During background deleting, the system directly compares the fingerprint without needing to read the disk data again. However, the algorithms used for fingerprint pre-calculation are usually computationally intensive and consume a lot of CPU resources, which directly leads to a significant increase in the latency of the data write path, seriously affecting the performance of front-end write operations.

[0006] Therefore, there is an urgent need in this field for a new data deduplication method that can simultaneously avoid large amounts of background data reading and expensive front-end fingerprint calculation, ensuring deduplication efficiency while minimizing interference with the read and write performance of the storage system. Summary of the Invention

[0007] Therefore, it is necessary to provide a fixed-length data deduplication method, apparatus, computer equipment, and computer program product to address the aforementioned technical problems.

[0008] Firstly, this application provides a fixed-length data deduplication method, the method comprising: Before each fixed-length data is written, each fixed-length data is divided into multiple sub-data blocks according to a preset segmentation mode. Multiple sampling data blocks are selected from the multiple sub-data blocks according to a preset sampling mode; The sub-fingerprints corresponding to each of the sampled data blocks are determined based on a preset hash algorithm; the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm. All the sub-fingerprints are combined according to a preset combination pattern to obtain similar fingerprints of the fixed-length data; Each of the fixed-length data and its similar fingerprints is written into a storage medium. During the background deleting phase, the similar fingerprints of each fixed-length data to be deleted are read from the storage medium, and the data is deleting based on the similarity of the read similar fingerprints.

[0009] In one embodiment, the step of performing data deduplication based on the similarity of the read similar fingerprints includes: Based on the similar fingerprints read, the current similarity threshold is dynamically calculated; Based on the current similarity threshold, the read similar fingerprints are grouped so that similar fingerprints with a similarity higher than the current similarity threshold are grouped together. Within the same group, deduplication is performed on fixed-length data with similar fingerprints that are to be deduplicated.

[0010] This scheme greatly reduces unnecessary disk reads and fingerprint comparison operations by using dynamic threshold grouping, strictly limiting the performance impact of background deduplication to a controllable range, and significantly reducing the system I / O and computing resource consumption of background deduplication.

[0011] In one embodiment, the step of dynamically calculating the current similarity threshold based on the read similar fingerprints includes: Calculate the overall similarity distribution among the similar fingerprints read; Calculate a preset statistic of the overall similarity distribution, and use the preset statistic as the current similarity threshold.

[0012] By dynamically determining the threshold based on the similarity of the data's own distribution, the grouping strategy can adapt to the characteristics of different batches of data to achieve the best grouping effect, avoid the unreasonable grouping problem that may be caused by a fixed threshold in different scenarios, and improve the adaptability and efficiency of the deduplication process.

[0013] In one embodiment, the step of grouping the read similar fingerprints includes: Based on the current similarity threshold, similar fingerprints with a similarity higher than the current similarity threshold are assigned to the same group using a locality-sensitive hashing algorithm or a clustering algorithm.

[0014] By employing efficient algorithms such as Locality Sensitive Hashing or clustering for grouping, it is possible to quickly complete the clustering of highly similar fingerprints, avoiding the huge overhead of calculating the pairwise similarity of all fingerprints, which is especially suitable for large-scale data scenarios.

[0015] In one embodiment, before each of the fixed-length data is written, the method further includes: Identify whether each of the fixed-length data is predefined special data; If there exists fixed-length data that is the special data, then a predefined fingerprint is directly assigned to it as a similar fingerprint and written together with it into the storage medium.

[0016] By providing a fast path for special data—that is, directly assigning predefined fingerprints and skipping all computation processes—zero computational overhead is achieved in writing such data, which specifically improves the peak write performance of the system when processing special data and further reduces the consumption of CPU resources.

[0017] In one embodiment, the special data includes data consisting entirely of 0s or entirely of 1s. All-0s or all-1s data is extremely common in storage systems and causes significant resource waste; this solution helps to specifically improve the write performance of such high-frequency special data.

[0018] In one embodiment, the preset segmentation mode is a uniform segmentation mode; and / or, the preset sampling mode is an equidistant sampling mode. This embodiment is easy to implement, simple and reliable, and helps to reduce the complexity of the data deduplication system.

[0019] Secondly, this application also provides a fixed-length data deduplication device, the device comprising: A similar fingerprint generation module is used to, before writing each fixed-length data, divide each fixed-length data into multiple sub-data blocks according to a preset segmentation mode; select multiple sample data blocks from the multiple sub-data blocks according to a preset sampling mode; determine the sub-fingerprint corresponding to each sample data block based on a preset hash algorithm; the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm; and combine all the sub-fingerprints according to a preset combination mode to obtain the similar fingerprint of the fixed-length data. A data writing module is used to write each of the fixed-length data and its similar fingerprints into a storage medium; The deduplication execution module is used to read similar fingerprints of each fixed-length data to be deduplicated from the storage medium during the background deduplication stage, and to perform data deduplication based on the similarity of the read similar fingerprints.

[0020] By proposing the above functional module architecture, it is helpful to reduce CPU overhead for write operations, reduce background I / O reads, and complete data deduplication intelligently and efficiently.

[0021] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a fixed-length data deduplication method.

[0022] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of a fixed-length data deduplication method.

[0023] One of the above technical solutions has the following advantages or beneficial effects: Since the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm, by converting the high-overhead computation of the full data into the preset hash computation of the sampled data blocks and combining them to generate similar fingerprints, the CPU computational overhead on the write operation path is fundamentally reduced; at the same time, in the background deduplication stage, by comparing lightweight similar fingerprints instead of reading the original data, the waste of I / O performance caused by reading non-duplicate data is effectively avoided; the above two aspects work together to isolate the impact of deduplication on performance in the background, and while ensuring the deduplication rate, the impact on the read and write performance of the storage system is minimized. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the principle of traditional background deleting. Figure 2 This is a schematic diagram illustrating the principle of another traditional backend deleting method. Figure 3 This is a flowchart illustrating a fixed-length data deduplication method in one embodiment of this application; Figure 4 This is a schematic diagram illustrating the principle of generating similar fingerprints before writing fixed-length data in one embodiment of this application. Figure 5 This is a schematic diagram illustrating the principle of similar fingerprint grouping during background deduplication in one embodiment of this application; Figure 6 This is a structural block diagram of a fixed-length data deduplication device in one embodiment of this application; Figure 7 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] It should be noted that the core concept of this application lies in transforming the full fingerprint calculation of fixed-length data into a lightweight calculation of sampled data blocks using a dimensionality reduction sampling approach. This results in the generation of similar fingerprints through combination, and then combined with intelligent grouping and comparison based on dynamic thresholds. This approach ensures a high deduplication rate while minimizing the impact of the deduplication process on the system's read and write performance. The specific implementation schemes of this application will be described below.

[0028] Figure 3 This is a schematic diagram illustrating the overall process of a fixed-length data deduplication method in one embodiment of this application. Figure 3 As shown, this method mainly includes two core stages: similar fingerprint generation during the data writing stage and intelligent grouping and deduplication during the background deduplication stage. The following section provides a detailed explanation of each step in conjunction with the flowchart.

[0029] S302, before writing each fixed-length data, divide each fixed-length data into multiple sub-data blocks according to the preset splitting mode.

[0030] In this step, fixed-length data can be understood as data blocks of fixed size, which are the basic units for data management in the storage system, such as 4KB, 8KB or 128KB; dividing each fixed-length data into sub-data blocks before writing is a prerequisite for achieving dimensionality reduction sampling.

[0031] The preset segmentation mode refers to the specific rules for segmentation. Optionally, the preset segmentation mode is a uniform segmentation mode, which divides fixed-length data into multiple sub-data blocks of the same size. For example, a 128KB fixed-length data can be uniformly divided into 16 8KB sub-data blocks. This segmentation method is simple to calculate and ensures that the position of each sub-data block in the original data is fixed, facilitating subsequent sampling and fingerprint generation. Of course, the preset segmentation mode can also be other segmentation methods, such as a segmentation mode based on data content features, for example, segmenting at the boundary where a significant change in data pattern is detected, so that the internal features of each sub-data block are more consistent; or an overlapping segmentation mode, that is, adjacent sub-data blocks contain partially overlapping data regions to capture potential feature information near the data block boundaries, etc., without specific limitations here.

[0032] S304, Select multiple sampling data blocks from multiple sub-data blocks according to the preset sampling mode.

[0033] This step aims to select a representative subset of sub-data blocks from all sub-data blocks to significantly reduce the amount of data required for subsequent fingerprint calculations. The preset sampling mode can be understood as a pre-defined sampling strategy. Optionally, the preset sampling mode can be an equal-interval sampling mode, where samples are selected from the segmented sub-data block sequence at fixed intervals; for example, from 16 sub-data blocks, one sample is selected every three, ultimately resulting in four sampled data blocks. This equal-interval sampling mode ensures that sampling points uniformly cover the entire data block, thereby preserving the global features of the original data to the greatest extent possible and avoiding feature concentration in a particular area. Furthermore, compared to full computation, sampling computation significantly reduces CPU overhead. Similarly, the preset sampling mode can also be set to other modes, such as a random sampling mode, where sub-data blocks are randomly selected using a pseudo-random number generator. This method helps avoid systematic biases caused by specific data patterns. It can also be a hierarchical sampling mode, where all sub-data blocks are divided into different layers, and samples are extracted from each layer to ensure the representativeness of different data regions. Alternatively, it can be configured as an adaptive sampling mode based on the importance of data content, such as assigning a higher sampling probability to sub-data blocks with higher data entropy values, etc. There are no limitations here.

[0034] S306, determine the sub-fingerprint corresponding to each sampled data block based on a preset hash algorithm; the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm.

[0035] The process of determining the sub-fingerprint corresponding to each sampled data block based on a preset hash algorithm involves calculating the fingerprint of each selected sampled data block. The preset hash algorithm, with lower computational overhead than a strong hash algorithm, can be understood as a non-encryption hash function that is fast in computation but has a relatively higher probability of different data producing the same hash value than a strong hash algorithm. Examples include, but are not limited to, CRC32, MurmurHash, or FNV algorithms. These algorithms are extremely fast and can meet the performance requirements of similarity comparison rather than precise comparison scenarios. By calculating weak hashes on sampled data blocks instead of the entire dataset, a second reduction in computational overhead can be achieved on the write path.

[0036] S308, combine all sub-fingerprints according to a preset combination pattern to obtain similar fingerprints with fixed-length data.

[0037] The preset combination modes include, but are not limited to, sequential concatenation or connection by delimiters. For example, assuming the sub-fingerprints of the four sampled data blocks are a1b2, c3d4, e5f6, and g7h8, the combined similar fingerprints could be a1b2c3d4e5f6g7h8, or a1b2-c3d4-e5f6-g7h8, etc. The feature vector generated by this combination method can represent the overall contour of the original fixed-length data and be used for subsequent similarity comparison.

[0038] Figure 4 The complete process from fixed-length data segmentation to generating similar fingerprints is shown. This scheme eliminates the need for the storage system to calculate hash fingerprints from the entire data. Instead, it calculates data blocks by sampling, and finally achieves deduplication through fingerprint comparison. This scheme can effectively reduce CPU consumption and improve the performance of the storage system.

[0039] Furthermore, the formula for generating similar fingerprints can be expressed as follows: ; ; in ，S finger Indicates similar fingerprints, n This indicates the number of sampled data blocks, usually an empirical value. FP 1 indicates the first 1 Fingerprint of each sampled data block, FP n Indicates the first n Fingerprint of each sampled data block, x n Indicates the first n One sampled data block, H(x n ) This indicates the default hash algorithm.

[0040] S310: Write each fixed-length data and its similar fingerprints to the storage medium, and in the background deleting stage, read the similar fingerprints of each fixed-length data to be deleted from the storage medium, and perform data deleting based on the similarity of the read similar fingerprints.

[0041] Each fixed-length data and its similar fingerprint are written to the storage medium, that is, the original data block and its corresponding similar fingerprint are persisted to the storage medium such as hard disk and SSD. Since the size of the similar fingerprint is much smaller than the original data, the additional storage overhead it brings can be ignored.

[0042] During the background deduplication phase, the system no longer reads the original data, but only reads lightweight similar fingerprints into memory for comparison. This similarity-based deduplication is an intelligent process; it doesn't directly compare whether fingerprints are completely identical, but rather performs deduplication based on the similarity of the read similar fingerprints. The similarity of similar fingerprints can be understood as a quantitative indicator measuring the closeness between two similar fingerprints, essentially comparing the structural matching degree of two similar fingerprints composed of sub-fingerprint sequences. Specifically, similarity can be obtained by calculating the proportion of identical sub-fingerprints at the same position in two similar fingerprints out of the total number of sub-fingerprints. For example, if two similar fingerprints are both composed of four sub-fingerprints FP1, FP2, FP3, and FP4 in sequence, and three of these sub-fingerprints are identical, then their similarity can be calculated as 75%. Of course, similarity can also be measured using other methods, such as the Jaccard similarity coefficient, which is determined by calculating the ratio of the intersection to the union of the sub-fingerprint sets in two similar fingerprints, or by calculating and normalizing the edit distance between two sub-fingerprint sequences. Regardless of the specific calculation method used, the fundamental purpose is to quickly and quantitatively screen out structurally similar fingerprints, providing a reliable candidate set for subsequent accurate deduplication operations.

[0043] The fixed-length data deduplication method provided in the above embodiments significantly reduces the CPU computation overhead on the data write path by converting the strong hash calculation of the full data into the weak hash calculation of sampled data blocks and combining them to generate similar fingerprints. This avoids the write operation performance bottleneck problem in traditional optimization schemes. Simultaneously, during the background deduplication stage, by comparing lightweight similar fingerprints instead of reading the original data, the I / O performance waste caused by reading non-duplicate data in traditional background deduplication schemes is effectively avoided. This application shifts the computational and I / O pressure from the system's write operations and front-end business to the background, thereby minimizing the impact on the read and write performance of the storage system while ensuring the deduplication effect.

[0044] In one embodiment, S310 specifically includes: dynamically calculating a current similarity threshold based on the read similar fingerprints; grouping the read similar fingerprints based on the current similarity threshold to aggregate similar fingerprints with a similarity higher than the current similarity threshold into the same group; and deleting fixed-length data to be deduplicated based on identical similar fingerprints within the same group. The dynamic calculation of the current similarity threshold can be understood as the threshold not being a fixed value, but rather calculated in real time based on the overall characteristics of the current batch of similar fingerprints, thereby enabling the grouping strategy to adapt to different data loads.

[0045] Grouping the read similar fingerprints can be understood as a coarse screening process, which aims to quickly gather a small number of fingerprints that are truly likely to be repeated from a large number of fingerprints, thereby greatly narrowing the scope of subsequent accurate comparison.

[0046] For fixed-length deduplication data with identical similar fingerprints, where identical similar fingerprints mean two fingerprints are completely identical, this precise comparison within a group minimizes the I / O overhead, even if disk data reading is required for final confirmation, because the scope has been significantly narrowed. This embodiment, through dynamic threshold determination, grouping, and final precise comparison within each group, minimizes unnecessary disk reads and fingerprint comparison operations while ensuring accuracy, strictly limiting the performance impact of background deduplication within a controllable range.

[0047] In one embodiment, the step of dynamically calculating the current similarity threshold based on the read similar fingerprints may specifically include: calculating the overall similarity distribution among the read similar fingerprints, calculating a preset statistic of the overall similarity distribution, and using the preset statistic as the current similarity threshold.

[0048] The overall similarity distribution can be understood as a statistical distribution of similarity values ​​obtained by analyzing the pairwise similarity between all similar fingerprints in the current batch, or by sampling estimation. The preset statistic is a pre-selected mathematical quantity used to extract representative values ​​from this distribution, such as the median, quantiles, or mode of the distribution. For example, the 75th percentile. Using the median means the system can automatically classify data relationships ranking in the top 50% as having high similarity, while using the 75th percentile is a more stringent selection method than the median. This embodiment dynamically determines the current similarity threshold based on the data's own distribution, enabling the system to intelligently match the appropriate threshold according to the similarity levels of different batches of data, thereby achieving the best grouping effect and avoiding the problems of unreasonable grouping and unexpected computational overhead caused by fixed thresholds in different scenarios.

[0049] In one embodiment, the step of grouping the read similar fingerprints may specifically include: based on the current similarity threshold, using a Locality Sensitive Hashing (LSH) algorithm or a clustering algorithm to assign similar fingerprints with a similarity higher than the current similarity threshold to the same group. LSH is an efficient algorithm for approximate nearest neighbor retrieval in high-dimensional data, ensuring that similar data points are mapped to the same hash bucket with a high probability; each bucket is considered a group, making this method extremely efficient. Clustering algorithms, such as DBSCAN, can naturally form groups based on the density of data points in high-dimensional space. Both algorithms efficiently achieve the goal of clustering highly similar fingerprints, avoiding the huge overhead of calculating the pairwise similarity of all fingerprints, and offering high accuracy and efficiency in grouping.

[0050] In one embodiment, before writing each fixed-length data, the fixed-length data deduplication method further includes: identifying whether each fixed-length data is predefined special data; if there is fixed-length data that is special data, then directly assigning a predefined fingerprint to it as its similar fingerprint so that it can be written to the storage medium together. In this embodiment, special data can be understood as data blocks with a fixed pattern that are common in the storage system; and a predefined fingerprint can be understood as a special identifier pre-assigned for this type of special data, such as assigning a fingerprint of 0x0000...00 to a block of all 0s, and assigning a fingerprint of 0xFFFF...FF to a block of all 1s, etc. This embodiment avoids high-cost hash calculations for known, fixed-result data by using a low-cost data content recognition step, thereby optimizing write performance for specific data write paths. This optimization is particularly effective when dealing with a large number of all-zero or all-one data blocks generated by the system. All-zero or all-one data blocks are extremely common in storage systems, such as in scenarios like disk pre-allocation and data erasure. Directly identifying and assigning special fingerprints to such data avoids redundant splitting, sampling, and hash calculations, resulting in considerable performance improvements and CPU cycle savings.

[0051] Of course, in some other embodiments, the special data may also include other data with a fixed padding pattern, such as data that is alternately padded with specific bytes, such as the 0xAA55AA55... pattern formed by alternating padding with bytes 0xAA and 0x55, or text data that is entirely padded with space characters, or fixed pattern data used for initialization under a specific file system format, such as the disk boot sector characteristic values ​​0xEB, 0x3C, 0x90, etc. of the FAT32 file system.

[0052] In some embodiments, the preset segmentation mode can be a uniform segmentation mode, and the preset sampling mode can be an equal-interval sampling mode. In this embodiment, the uniform segmentation mode and the equal-interval sampling mode are easy to implement, simple and reliable, and the effect is predictable. In some other embodiments, the preset hash algorithm can be selected as CRC32, MurmurHash, or FNV algorithm, etc. These algorithms are all classic hash algorithms with fast calculation speed and controllable collision rate. The schemes of these embodiments have strong practicality and repeatability.

[0053] In a specific embodiment of this application, please refer to Figure 4 and Figure 5 Taking a fixed-length data set of 128KB as an example, the fixed-length data deduplication method can be as follows: First, the fixed-length data can be evenly divided into 16 8KB sub-data blocks. Four sampled data blocks are selected by sampling at equal intervals according to a 1 / 4 sampling rate. For each sampled data block, a preset hash algorithm is applied to calculate a sub-fingerprint, such as FP1, FP2, FP3, and FP4. Finally, a similar fingerprint of the data is generated by string concatenation, i.e., S. finger =FP1FP2FP3FP4, and then write the fixed-length data and its similar fingerprints into the storage medium; at the same time, for special data such as all 0 or all 1 data blocks, the system directly identifies and assigns the corresponding special fingerprint, thereby skipping all fingerprint calculation processes.

[0054] During background deleting, in response to the system reading a batch of similar fingerprints from a fixed-length dataset, the overall similarity distribution is calculated, and the 75th percentile is set as the dynamic threshold (i.e., ...). Figure 5 The algorithm first uses a locality-sensitive hashing (LSH) method to quickly group fingerprints with similarity exceeding a certain threshold into the same group. Then, within that group, it precisely compares data blocks with identical fingerprints and performs deduplication. This example scheme, through the synergistic effect of segmentation sampling, dynamic grouping, and direct fingerprint attachment based on special data identification, achieves significant performance improvements under real-world system loads, maintaining a high deduplication rate while reducing write latency and background I / O usage.

[0055] In summary, the fixed-length data deduplication method proposed in the embodiments of this application has at least the following beneficial effects: 1. The implementation scheme calculates the similar fingerprint of data by dimensionality reduction sampling. By splitting fixed-length data into a certain number of data blocks, performing weak hash calculation on the sampled data blocks, and finally generating similar fingerprints for fixed-length data, the performance of write operations can be improved, thereby improving the deduplication performance.

[0056] 2. The implementation scheme performs deduplication operations based on the similarity of similar fingerprints. Only data with high similarity can be compared and then deduplicated. This can effectively reduce the impact of background deduplication on the entire system, improve the performance of read and write operations, and thus improve deduplication performance.

[0057] 3. The embodiment scheme uses a hybrid fingerprint collection method to collect special similar fingerprints for special data, avoids hash calculations, improves the performance of write operations, and thus improves deduplication performance.

[0058] The deficiencies of the above solutions and the proposed solutions are the result of the inventors' practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventors' contributions to this disclosure.

[0059] It should be understood that, for the foregoing method embodiments, although the steps in the flowcharts are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the method embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0060] Based on the same inventive concept, this application also provides a fixed-length data deduplication device for implementing the fixed-length data deduplication method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the fixed-length data deduplication device provided below can be found in the limitations of the fixed-length data deduplication method described above, and will not be repeated here.

[0061] In one embodiment, such as Figure 6 As shown, a fixed-length data deduplication device 600 is provided, comprising: The similar fingerprint generation module 601 is used to divide each fixed-length data into multiple sub-data blocks according to a preset segmentation mode before writing each fixed-length data; select multiple sample data blocks from the multiple sub-data blocks according to a preset sampling mode; determine the sub-fingerprint corresponding to each sample data block based on a preset hash algorithm; the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm; and combine all the sub-fingerprints according to a preset combination mode to obtain the similar fingerprint of the fixed-length data. The data writing module 602 is used to write fixed-length data and their similar fingerprints into the storage medium; The deduplication execution module 603 is used to read the similar fingerprints of each fixed-length data to be deduplicated from the storage medium during the background deduplication stage, and to perform data deduplication based on the similarity of the read similar fingerprints.

[0062] In one embodiment, the deduplication execution module 603 is specifically used to: dynamically calculate the current similarity threshold based on the read similar fingerprints; group the read similar fingerprints based on the current similarity threshold to aggregate similar fingerprints with a similarity higher than the current similarity threshold into the same group; and perform deduplication on the fixed-length data to be deduplicated that have the same similar fingerprints within the same group.

[0063] In one embodiment, the deduplication execution module 603 is further configured to: calculate the overall similarity distribution among the read similar fingerprints; calculate a preset statistic of the overall similarity distribution, and use the preset statistic as the current similarity threshold.

[0064] In one embodiment, the deduplication execution module 603 is further configured to: based on the current similarity threshold, assign similar fingerprints with a similarity higher than the current similarity threshold to the same group using a locality-sensitive hashing algorithm or a clustering algorithm.

[0065] In one embodiment, the similar fingerprint generation module 601 is further configured to: identify whether each fixed-length data is predefined special data; if there is fixed-length data that is special data, then directly assign a predefined fingerprint to it as its similar fingerprint and write it together with it into the storage medium.

[0066] In one embodiment, special data includes data that is all 0s or all 1s.

[0067] In one embodiment, the preset segmentation mode is a uniform segmentation mode; And / or, the preset sampling mode is equal-interval sampling mode.

[0068] Each module in the aforementioned fixed-length data deduplication device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] Furthermore, in the above-described implementation of the fixed-length data deduplication device, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the fixed-length data deduplication device can be divided into different program modules to complete all or part of the functions described above.

[0070] In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, and a communication interface. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a fixed-length data deduplication method.

[0071] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0072] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0073] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0074] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0075] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0076] The terms “comprising” and “having”, and any variations thereof, in the embodiments herein are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or (module) units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0077] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A fixed-length data deduplication method, characterized in that, include: Before each fixed-length data is written, each fixed-length data is divided into multiple sub-data blocks according to a preset segmentation mode. Multiple sampling data blocks are selected from the multiple sub-data blocks according to a preset sampling mode; The sub-fingerprints corresponding to each of the sampled data blocks are determined based on a preset hash algorithm; The computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm; All the sub-fingerprints are combined according to a preset combination pattern to obtain similar fingerprints of the fixed-length data; Each of the fixed-length data and its similar fingerprints is written into a storage medium. During the background deleting phase, the similar fingerprints of each fixed-length data to be deleted are read from the storage medium, and the data is deleting based on the similarity of the read similar fingerprints.

2. The method according to claim 1, characterized in that, The step of performing data deduplication based on the similarity of the read similar fingerprints includes: Based on the similar fingerprints read, the current similarity threshold is dynamically calculated; Based on the current similarity threshold, the read similar fingerprints are grouped so that similar fingerprints with a similarity higher than the current similarity threshold are grouped together. Within the same group, deduplication is performed on fixed-length data with similar fingerprints that are to be deduplicated.

3. The method according to claim 2, characterized in that, The step of dynamically calculating the current similarity threshold based on the read similar fingerprints includes: Calculate the overall similarity distribution among the similar fingerprints read; Calculate a preset statistic of the overall similarity distribution, and use the preset statistic as the current similarity threshold.

4. The method according to claim 3, characterized in that, The step of grouping the read similar fingerprints includes: Based on the current similarity threshold, similar fingerprints with a similarity higher than the current similarity threshold are assigned to the same group using a locality-sensitive hashing algorithm or a clustering algorithm.

5. The method according to any one of claims 1 to 4, characterized in that, Before writing each of the fixed-length data, the method further includes: Identify whether each of the fixed-length data is predefined special data; If there exists fixed-length data that is the special data, then a predefined fingerprint is directly assigned to it as a similar fingerprint and written together with it into the storage medium.

6. The method according to claim 5, characterized in that, The special data includes data whose contents are all 0s or all 1s.

7. The method according to any one of claims 1 to 4, characterized in that, The preset segmentation mode is a uniform segmentation mode; And / or, the preset sampling mode is equal-interval sampling mode.

8. A fixed-length data deduplication device, characterized in that, include: A similar fingerprint generation module is used to divide each fixed-length data into multiple sub-data blocks according to a preset segmentation mode before each fixed-length data is written. Multiple sampling data blocks are selected from the multiple sub-data blocks according to a preset sampling mode; The sub-fingerprints corresponding to each of the sampled data blocks are determined based on a preset hash algorithm; the computational overhead of the preset hash algorithm is lower than that of the strong hash algorithm; all the sub-fingerprints are combined according to a preset combination pattern to obtain similar fingerprints of the fixed-length data; A data writing module is used to write each of the fixed-length data and its similar fingerprints into a storage medium; The deduplication execution module is used to read similar fingerprints of each fixed-length data to be deduplicated from the storage medium during the background deduplication stage, and to perform data deduplication based on the similarity of the read similar fingerprints.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images