Methods for implementing deduplication and corresponding storage devices

Abstract

The present invention relates to a method for implementing deduplication and a corresponding storage device. Block processing is performed on a file, and the fingerprint of each data block is calculated. The fingerprint of each data block is checked in a pre-stored fingerprint library to confirm whether it exists. If the fingerprint of the data block is in the pre-stored fingerprint library, the index of the data block is saved and the corresponding data block is abandoned. If the fingerprint of the data block is not in the pre-stored fingerprint library, the index of the data block is saved, and a preset encoding process is performed on the data block and the encoded data is stored in a server.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of data storage, and more specifically, to a method for implementing deduplication in data storage, a corresponding system, and a corresponding storage device. Background Art

[0002] As the amount of data grows explosively, storage capacity has grown from GB to TB or even PB. It is well known that big data has penetrated all walks of life, and its most notable feature is the huge volume of data. For example, the amount of data created and copied in the world in a single year exceeds trillions of GB. The storage cost of data for enterprises and government agencies will become increasingly high, but research has found that nearly 70% of certain categories of data are duplicated. The existence of duplicate data not only wastes storage space, but more importantly, it poses great challenges to data processing speed and calculation accuracy.

[0003] Taking the automobile industry as an example, if 1GB of data is generated per second based on 600 hours of driving per year, then driverless cars will generate an average of about 2PB of data per year. Vehicle recorders can record video data of the vehicle driving process and the driver's control data of the vehicle to help the driver understand the driving conditions and provide control evidence, but the huge amount of data means that the old data has to be overwritten with the new data and some important information is lost. In fact, the old data that is overwritten includes key data that should be retained and more unnecessary duplicate data.

[0004] If the files or data blocks in the storage system are deduplicated, only a limited number of duplicates or a single copy of the duplicate data is retained and the rest of the duplicate data is deleted, the storage capacity will be greatly reduced. In order to reduce the large amount of duplicate and redundant data in the storage system, deduplication technology has become a hot topic in recent years.

[0005] Deduplication is a technology widely used in storage systems. It can greatly reduce the storage space occupied by analyzing whether there are duplicates in the data and retaining only one copy of the same data. Deduplication technology can be classified in different ways from different dimensions. According to the timing of occurrence, it can be divided into two categories: real-time deduplication and offline deduplication. Real-time deduplication: During the data writing process, it detects in real time whether the new data is duplicated with the old data that has been stored, removes the duplicated parts, and only writes the newly added parts. Offline deduplication: Data is first written directly to the storage area, and then the data is analyzed to find the duplicated parts, delete them and release the space.

[0006] Simply pursuing deduplication may not be the best choice. For example, in some cases, the deduplication function is used only from the perspective of redundancy reduction, so there is still room for improvement in the release and improvement of storage space. Typically, the deduplication function highlights this feature in the all-flash market. At this time, the use of deduplication alone will be very limited in terms of the significant release of storage space and the improvement of the useful area of ​​storage space, and the redundancy reduction of storage space still needs to be improved. Summary of the invention

[0007] The present application relates to a method for implementing deduplication, which is characterized by comprising:

[0008] Perform block processing on the file and calculate the fingerprint of each data block;

[0009] Check the fingerprint of each data block against the pre-stored fingerprint database to confirm whether it exists:

[0010] If yes, the index of the data block is saved and the corresponding data block is discarded;

[0011] If not, the index of the data block is saved, and a preset encoding process is performed on the data block, and the encoded data is stored in the server.

[0012] The above method: in the block processing stage, a specified scale is set, the file to be processed is segmented according to the specified scale to obtain a series of data blocks, and the fingerprint of each data block is calculated based on the specified scale.

[0013] The above method: in the block processing stage, the file to be processed is divided into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale.

[0014] The above method: when checking the fingerprint of a data block in the fingerprint library, a HASH search method is used to search whether the fingerprint of the data block exists in the fingerprint library.

[0015] The above method: the encoding process includes the encoding process of the error correction code type. When the data is restored, the encoded data is extracted according to the index of the encoded data and its original data block is reconstructed according to the decoding principle of the error correction code.

[0016] The above method: repeatedly reconstruct all the encoded data until a series of original data blocks corresponding to all the encoded data are obtained, and then they are restored and combined into the original file.

[0017] In the above method, the coded data is compressed before being stored in the server. When the data is restored, the compressed coded data is extracted according to the index of the coded data and the original coded data is decompressed.

[0018] The above method: the preset coding process includes coding process using any one of the erasure codes of cascaded low-density erasure coding, digital fountain code coding or Reed-Solomon coding.

[0019] The above method: the way of checking the fingerprint of each data block includes K-shingle fingerprint algorithm or Simhash fingerprint algorithm or Minhash fingerprint algorithm or longest sentence signature algorithm.

[0020] The above method: data compression includes lossless compression using at least one of Huffman coding, arithmetic coding, run length coding or string table compression algorithms.

[0021] The above method: data compression includes lossy compression using at least one of predictive coding, transform coding and hybrid coding.

[0022] The above method: in the block processing stage, the data block fingerprint is calculated based on the elastic scale, and the file to be processed is divided into a series of data blocks with different scales. The larger the elastic scale when the series of data blocks are divided, the smaller the data ratio of the check block used in the encoding processing relative to the data to be encoded.

[0023] The present application also relates to a storage device for implementing deduplication, characterized in that it includes:

[0024] A data segmentation module is used to perform block processing on the file to obtain segmented data blocks;

[0025] A judgment module is used to extract the fingerprint of each data block and check the fingerprint of each data block in a pre-stored fingerprint library to confirm whether the fingerprint of the data block exists in the fingerprint library;

[0026] The encoding module implements a preset encoding process on the data block if the judgment module believes that the fingerprint of the data block is not in the fingerprint library; on the contrary, if the judgment module believes that the fingerprint of the data block is already in the fingerprint library, the encoding module does not implement the preset encoding process on the data block and abandons the corresponding data block.

[0027] The above-mentioned storage device for implementing deduplication, wherein: the encoding processing performed by the encoding module includes encoding processing of the error correction code type, extracting the encoded data during data restoration and reconstructing its original data blocks based on the decoding principle of the error correction code; and repeatedly reconstructing all the encoded data until a series of original data blocks corresponding to all the encoded data are obtained; and the recovery module of the storage device recombines the original data blocks to restore them into the original file.

[0028] The storage device for implementing deduplication mentioned above, wherein: before the encoded data is stored in the server or storage device, data compression is first performed and when the data is restored, the compressed encoded data is extracted and the original encoded data is decompressed.

[0029] In the storage device implementing deduplication, the data segmentation module sets a specified scale during the block processing phase and segments the file to be processed according to the specified scale to obtain a series of data blocks, and calculates the fingerprint of each data block based on the specified scale.

[0030] In the storage device implementing deduplication, the data segmentation module segments the file to be processed into a series of data blocks of different scales during the block processing phase, and calculates the fingerprint of each data block based on the elastic scale.

[0031] In the storage device implementing deduplication, when the judgment module verifies the fingerprint of a data block in the fingerprint library, a hash search method is used to search whether the fingerprint of the data block exists in the fingerprint library.

[0032] In the storage device for implementing deduplication, the fingerprint extraction method of the judgment module includes a K-shingle fingerprint algorithm, a Simhash fingerprint algorithm, a Minhash fingerprint algorithm, or a longest sentence signature algorithm.

[0033] In the above-mentioned storage device for implementing deduplication, data compression includes lossless compression using any one of Huffman coding, arithmetic coding, run-length coding or string table compression algorithms.

[0034] In the above-mentioned storage device for implementing deduplication, data compression includes lossy compression using any one of predictive coding, transform coding and hybrid coding.

[0035] The storage device for implementing deduplication described above divides the file to be processed into a series of data blocks of different scales during the block processing stage, and calculates the data block fingerprint based on the elastic scale. The larger the elastic scale when dividing the series of data blocks, the smaller the data ratio of the check block used in the encoding processing relative to the data to be encoded. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the above purposes, features and advantages more understandable, the following text will explain the specific implementation methods in detail in conjunction with the drawings. After reading the following description and referring to the following drawings, the features and advantages of the present application will become apparent.

[0037] Figure 1 There is a large amount of duplicate content or useless redundant data to be deleted in the storage space.

[0038] Figure 2 It is the general framework of segmentation, judgment, and encoding used by storage devices to implement deduplication.

[0039] Figure 3 It is the segmentation, judgment, encoding and compression framework used by storage devices to implement deduplication.

[0040] Figure 4 This is an example of block processing, fingerprint determination, and encoding used in implementing deduplication methods.

[0041] Figure 5 It is a method to implement deduplication using block processing, fingerprint judgment, compression and encoding.

[0042] Figure 6 It is an example of removing duplicate content or useless redundant data from storage space to be deleted. DETAILED DESCRIPTION

[0043] The technical solutions disclosed in the present application will be clearly and completely described below in conjunction with specific embodiments. However, the embodiments described are only embodiments used for narrative explanation in the present application and are not all embodiments. Based on these embodiments, technicians in this field should realize that any solution obtained without creative work falls within the scope of protection of the present application.

[0044] See also Figure 1 For deduplication, the main process is to extract the HASH fingerprint of the data from the storage system such as the file system or database and put it into the host memory or a dedicated system for data comparison during the data delivery process. The duplicate data is deleted and the index is marked based on the comparison results. For non-duplicate data, it is added to the HASH fingerprint library and the processed data is then delivered to the data storage device to achieve the purpose of removing duplicate data.

[0045] See also Figure 1 , describes the spatial distribution inside the storage. In addition to the normal data D0, it is also found that there are redundant data or duplicate data D1-D3 in the storage. The useful normal data D0 should be retained and its security and reliability should be paid attention to. For example, when the data disk is set to use shared storage on the server, the storage only needs to check whether the file is locally on the data disk when each file is stored. Generally speaking, if the file exists locally on the data disk, it should not be stored. On the contrary, if the file does not exist locally on the data disk, it is allowed to be stored. This is an assumption for filtering duplicate data.

[0046] See also Figure 1 , multiple copies of data D1 are duplicate data, multiple copies of data D2 are duplicate data, and the remaining multiple copies of data D3 are duplicate data. Trying to establish a rule for filtering duplicate data for each type of duplicate data will complicate deduplication, because the limited D1-D3 are only a small range of examples among millions of duplicate data. Obviously, this method is not applicable to large-capacity storage systems.

[0047] See also Figure 1 , the total amount of duplicate data D1-D3 is recorded as R-DATA, and the loss rate is S-LOSS. The loss rate is equal to the total amount of duplicate data divided by the total storage capacity of the storage system. It is worth noting that the limited D1-D3 in the figure is only a small example among millions of duplicate data, and the loss rate S-LOSS increases with the growth of R-DATA.

[0048] See also Figure 1 If the duplicate data D1-D3 are of file type, it is easy to know and query whether the data is stored in the data disk. If they are of block storage type, it is difficult to directly know and query whether the data is stored in the data disk. Therefore, it is very necessary to implement deduplication, that is, de-duplication of data.

[0049] See also Figure 1 The key point of the deduplication function is that only one copy of the data block is retained in the deduplication library that stores the data, and each copy of the data block is different and unique. The data backed up using the deduplication function is different from the non-deduplication data. Note that the data in the non-deduplication backup set is independent of each other, and can be directly deleted without affecting the data in other backup sets. After using the deduplication function, it is natural that the following phenomenon will occur: when multiple backup sources perform backups, after using the deduplication function to divide the blocks, it will be found that some of the data is the same, that is, the data blocks stored in the deduplication library may be referenced by multiple backup sets. According to the requirements of deduplication, only a single copy of the data can be retained for reference by multiple backup sources.

[0050] See also Figure 1 The usual practice of the deletion function is to record the number of references of each data block in the deduplication database. When deleting a backup set, find each data block used in the backup set and subtract the number of references from the corresponding number of times used. If the number of references is zero, it means that the block is no longer used by the backup set and can be completely deleted. This approach has the following characteristics: First, when recording the number of indexes of each block, the granularity is relatively fine. Every time a backup task is executed, whether the data block is a new block or not, the database needs to be accessed (for example, a new fingerprint record needs to be inserted for a new block and the number of references for an existing block needs to be updated). This may affect the performance of the backup. Even if the deletion operation is completed, the space in the deduplication database is not released (the data block on the disk cannot be truly cleaned if the number of references is not zero). Secondly, when deleting a backup set, since all the deduplication blocks used in the backup set need to be traversed and the fingerprint tables corresponding to all the blocks used need to be adjusted, the deletion task will take a long time and the amount of data is extremely large. Therefore, the existing deletion scheme needs to be improved in performance and adjusted to be suitable for application scenarios where backup and deletion are frequent.

[0051] See also Figure 2 , the deletion of duplicate data D1-D3 is Figure 2 It is reflected in the . It is worth noting that for all-flash storage, which has high response requirements, fixed-length deduplication can be used with high speed. For storage such as archiving and backup, variable-length deduplication can be used with a high deduplication rate to save costs. This will be explained in detail below. In addition, given that in some cases, the use of the deduplication function is only considered from the perspective of redundancy reduction, there is still room for improvement in the release and improvement of storage space. For example, the deduplication function highlights this feature in the all-flash market. At this time, the combination of deduplication and compression will be more meaningful for the substantial release of storage space and the improvement of the useful area of ​​storage space.

[0052] See also Figure 2 , the deduplication domain applicable to the present application can be a logical area where deduplication takes effect, that is, a data storage area where data deduplication is to be performed. In the case of real-time deduplication, in the same deduplication domain, the data to be written is segmented according to a specific algorithm and fingerprints are calculated. The fingerprints are compared to determine whether the same data segment exists. The same data segment is only stored in one deduplication domain to reduce storage space usage. In the case of offline deduplication, in the same deduplication domain, the fingerprints in the fingerprint library are compared to determine whether the same data segment exists in the stored data. If the same data segment exists, the redundant data segment is deleted, so that only one copy of the same data segment is stored in one deduplication domain to reduce storage space usage. This process is applicable to the context example of the present application.

[0053] See also Figure 2 The storage device at least includes a data segmentation module 101, a judgment module 102, and an encoding module 104. The functions and specific operation processes of the aforementioned modules will be explained one by one below. It is worth noting that the functional modules of the storage device include but are not limited to the parts shown in the figure. All kinds of storage technology solutions and storage functions applicable to memory in the industry can be applied to this storage device. Different architectures such as X86 or LOONGARCH do not affect the ability to transplant general storage technologies under different architectures to this storage device.

[0054] See also Figure 2, about the data segmentation module 10: it is used to perform block or segmentation processing on files (for example, files to be deduplicated or files to be compressed or files that take into account both deduplication and compression tasks) to obtain segmented data blocks (Data Block). As mentioned above, if the duplicate data D1-D3 belongs to the file type, it can be easily checked whether there is duplicate data. If not, the duplicate data D1-D3 does not belong to the file type, then the duplicate check will become difficult and it will be difficult to substantially delete the duplicate data D1-D3. The meaning of the data block (Data Block) is just like this, that is, to query duplicate data and perform deduplication tasks at a finer granularity.

[0055] See also Figure 2 One of the algorithms for dividing or splitting a file into blocks to obtain data blocks is a full file block algorithm, for example, the whole file is treated as a complete data block for block processing. This is an optional implementation case of the data segmentation module 10 performing block processing on the file, but full file block processing is not optimal.

[0056] See also Figure 2 The second algorithm for obtaining data blocks by dividing or segmenting a file is a fixed size chunking algorithm, which sets a fixed file chunk size before splitting the file and splits the file according to the size. This is an optional example of the data segmentation module 10 for chunking the file.

[0057] See also Figure 2 The third algorithm for obtaining data blocks by dividing or segmenting or chunking a file is a content-based chunking algorithm such as a CDC (Content Defined Chunking) algorithm. This type of chunking can usually use a sliding window technique to determine the boundary by calculating a window such as a RABIN fingerprint. Typically, in order to solve the contradiction between improving the data reduction rate and reducing the additional metadata storage overhead, the industry has proposed a content-based Finger-diff algorithm. This is an optional implementation case of the data segmentation module 10 performing chunking processing on a file.

[0058] See also Figure 2 The fourth algorithm for obtaining data blocks by dividing or splitting a file is a finer-grained division, such as using a byte / bit level division and performing a deduplication task. This is an optional implementation case of the data segmentation module 10 performing block processing on a file.

[0059] See also Figure 2, about the judgment module 102: it is used to extract the fingerprint of each data block, and check the fingerprint of each data block in the pre-stored fingerprint library 103 to confirm whether the fingerprint of the data block exists in the fingerprint library 103. Assuming that a total of about N data blocks are separated, the fingerprint calculated for the first data block is used to confirm whether the fingerprint of the first data block exists in the fingerprint library 103, and the fingerprint calculated for the second data block is used to confirm whether the fingerprint of the second data block exists in the fingerprint library 103, and the fingerprint calculated for the Nth data block is used to confirm whether the fingerprint of the Nth data block exists in the fingerprint library 103. For example, if the fingerprint of the Kth data block exists in the fingerprint library 103, it is confirmed that this data block is a duplicate data block. Positive integer K≤N, each data block is checked one by one.

[0060] See also Figure 2 The basic operation principle of the judgment module 102 is as follows: first, the file to be analyzed can be divided into data blocks using the file segmentation method, and fingerprints are calculated for each of the segmented data blocks. Then, the fingerprints are used as keywords to check in the fingerprint library 103, for example, using HASH search. If a matching fingerprint value of a data block is found in the fingerprint library 103, it means that the data block is a duplicate data block, for example, only the index number of the data block is stored. Otherwise, it means that the current checked data block is a new data block, and the data block is stored and corresponding meta information is created.

[0061] See also Figure 2 Therefore, we can know that deduplication is a storage capacity optimization technology. It eliminates duplicate data in the storage system and reduces the data actually stored in the system or transmitted over the network. It has been widely used in backup, long-term archiving and data disaster recovery. With the development of software-defined storage and the emergence of virtualization technology, dedicated processor technology and new storage media, combined with online duplicate data processing and reducing the unit capacity cost of storage, the demand for deduplication technology is becoming more and more urgent.

[0062] See also Figure 2 Regarding the encoding module 104, if the judgment module 102 believes that the fingerprint of a data block such as the K-th data block is not in the fingerprint library 103, the encoding module 104 implements a preset encoding process on the data block such as the K-th data block; on the contrary, if the judgment module 102 believes that the fingerprint of a data block such as the K-th data is in the fingerprint library 103, the encoding module 104 does not implement the preset encoding process on the data block such as the K-th data, and abandons the corresponding data block.

[0063] See also Figure 2In terms of all-flash storage, the latest data from Gartner shows that the global external storage market grew by 0.5% year-on-year in the first quarter of 2023, of which all-flash arrays grew by 3.6% year-on-year, and the market size exceeded that of non-all-flash arrays, accounting for 50.4% of the entire external storage market revenue. In the past few years, the global economy and market have experienced great turbulence and changes. It is commendable that all-flash storage can still cross cycles and maintain growth. At the same time, as the digital transformation of thousands of industries has entered the deep water zone, data has become the core element, strategic resource and foundation of value creation in the digital age. New applications such as big data, machine learning, and large models are booming, bringing new data storage requirements such as PB, tens of millions of IOPS and intelligent management, and also accelerating the arrival of an all-flash era.

[0064] See also Figure 2 , and fast response is required for the All Flash Storage type. In the stage of performing block processing, a specified scale is set, and the file to be processed is segmented according to the specified scale to obtain a series of data blocks, and the fingerprint of each data block is calculated based on the specified scale. The specified scale can essentially be classified as a fixed-size block algorithm. In the stage of segmenting files, a specified or fixed file block size is first set and the file needs to be segmented and segmented according to this file block size. Fixed-length deduplication is very suitable for all-flash storage and meets the requirements of high speed and fast response required by all-flash storage. Based on the specified scale means that in the series of data blocks segmented from the file: the data scale or data size of any two different data blocks is exactly the same.

[0065] See also Figure 2 For archiving and backup storage needs, unlike fast-read and fast-store flash memory, in the block processing stage, the file to be processed is divided into a series of data blocks of different sizes, and the fingerprint of each data block is calculated based on the elastic scale. The elastic scale refers to the variable size of the scale. Based on the elastic scale, it means that in a series of data blocks divided from the file: the data scale or data size of any two different data blocks can be the same or different; even in a series of data blocks, the data scale or data size of all data blocks are different.

[0066] See also Figure 2, elastic scale can adopt the block technology of variable size data blocks, for example, divide the file into data blocks of different sizes based on the file content. The fingerprint value of the data content is calculated using the Rabin fingerprint method. Rabin fingerprint is the current efficient fingerprint calculation function. Using the randomness of the HASH function, it shows uniformly distributed feature points for the calculation results of any data. The specific scheme of variable size will be explained in detail below.

[0067] See also Figure 2 , for example, content-based data block partitioning methods: pre-set a pair of integers D, r (D>r) and set a specified width l of a sliding window (such as r=D-1). For a sequence S=s1, s2, ...sn, if and only if the edge of the window stops at a certain k position, there is a subsequence W=s(k-l+1), s(k-l+2), ...sk, and the fingerprint function calculation result of the subsequence is h(W)mod D=r, then there is a D-match at the k position. In fact, the k position is also the boundary position of a corresponding data block. The block partitioning method based on the sliding window belongs to the scope of the prior art.

[0068] See also Figure 2 For storage such as archiving and backup, variable length deduplication (such as dividing the file to be processed into a series of data blocks of different scales, and calculating the fingerprint of each data block based on the elastic scale) can be used to achieve the beneficial effects of high deduplication rate and cost saving. In actual operation, starting from the file header, the data in the fixed-size but overlapping sliding windows are used as subsequences of Rabin fingerprints, and the fingerprint of each window position is calculated. When the fingerprint condition is met, the boundary of the window position at this time is used as the block boundary. Repeat this process until the entire file data is divided into a series of data blocks in a sliding window method. Then, the HASH value of each divided data block is calculated using a HASH function or MD5 or SHA, and they are managed and stored in the HASH value library. Divide into data blocks according to the above method and compare the HASH value of the data block with the HASH value of the stored data block. If the same value is verified, it means that it is a data block that needs to be deleted. Otherwise, the data block is retained and the HASH value library information is updated.

[0069] See also Figure 2 When the judgment module 102 checks the fingerprint of the data block in the fingerprint library 103, it uses the HASH search method to search whether the fingerprint of the data block already exists in the fingerprint library 103.

[0070] See also Figure 2The key reason why the encoding module 104 is introduced into the deduplication system or storage device is that the deduplication system does not save the data file as a whole when saving data. As mentioned above, the data file is divided into multiple data blocks according to a certain segmentation algorithm. In fact, this breaks the characteristics of the file itself. The potential danger is that the original file will lose information in batches or partially lose information. Obviously, this is not expected to happen. The advantage of dividing a data file into multiple data blocks is the convenience of deduplication. The deep-seated problem that cannot be ignored is that the data blocks are shared by all data files in the storage system or device. In other words, the data block not only affects itself, but any data block may affect other files that have a sharing mechanism with it. Imagine that if one or several data blocks are lost or damaged after segmentation, it will affect the restoration of multiple files, which undoubtedly reduces the data security of the storage system. The deduplication system naturally has the potential threat of batch or partial data loss.

[0071] See also Figure 2 , if the judgment module 104 believes that the fingerprint of the data block is not in the fingerprint library, it will perform a preset encoding process on the data block. Introducing encoding in the deduplication system can improve the data security of the deduplication system to a certain extent, because the preset encoding can restore the real data through the corresponding decoding. Even if information loss occurs, the original real data can be inferred through data decoding. The preset encoding can use erasure code encoding. The basic principle of erasure code is to create a mathematical function to describe a set of numbers, and then check their accuracy. Once a number is lost, it can also be corrected and restored. Polynomial interpolation or what the industry calls oversampling is a key technology used in erasure codes.

[0072] See also Figure 2 , the encoding module 104 involves two parts of data, one is the initial data to be encoded, such as the data blocks obtained by segmentation, and the other is the additional or redundant data that needs to be added to provide protection to avoid data loss. Compared with the traditional scheme, the present application adds a part of additional or redundant data. The additional or redundant data used for error correction means that they will also consume part of the storage space of the storage device. Erasure codes are also called forward error correction (FEC) codes, and different types of erasure codes are generated. One of the typical and more reliable types is RS (Reed-Solomon), which can be reconstructed using any combination of k symbols or data blocks, even if m symbols are lost or unavailable.

[0073] See also Figure 2 , the preset coding process of the coding module 104 can use the coding process of the cascaded low-density erasure code class or the coding process of the digital fountain code class. Comparison of different codes. The cascaded low-density erasure code is a certain erasure code technology inherited from the development of the parity check code. The encoding and decoding process uses a simple binary XOR operation and has the advantages of simple operation, error-prone and not consuming too much computer resources, but has the disadvantage of long-distance segmentation defects in terms of the maximum distance segmentation property. RS coding involves a large number of matrix operations, with high accuracy, but the encoding and decoding process is much slower than the cascaded low-density erasure code. The digital fountain code is an erasure code with unlimited code rate, which is often referred to as rateless coding in the industry. Its principle is that the sequence of coded symbols generated from the encoding of k source symbols is infinite, and each coded symbol is equal to the XOR sum of multiple randomly selected source symbols. The receiver can decode and restore all k source symbols by receiving any m symbols. The coding process of the coding module 104 includes the coding process of the error correction code class, and the type of error correction code is not limited to a specific type. When restoring the data, the encoded data is extracted according to the index of the encoded data and its original data block is reconstructed according to the decoding principle of the error correction code. For example, the index of the Kth data block is saved and the Kth data block is subjected to a preset encoding process to obtain the encoded data en_data_K, which is stored in a server or a similar memory or storage device. When restoring, the encoded data en_data_K is extracted according to the index of the encoded data en_data_K and its original data block is reconstructed according to the decoding of the error correction code.

[0074] See also Figure 3 As mentioned above, to avoid data loss, additional or redundant data needs to be configured. Compared with the traditional solution, some additional or redundant data will be added. The storage space occupied by the redundant data causes waste. In essence, it is a compromise to sacrifice part of the storage space by improving the reliability of the data. In order to solve this problem, this application introduces data compression into the deduplication system based on the preset encoding process to reduce the waste of storage space.

[0075] See also Figure 3, the introduction of the compression module 105 can partially reduce the waste of storage space. The coded data is compressed before being stored in the server. When the data is restored, the compressed coded data is extracted according to the index of the coded data and the original coded data is decompressed. For example, before the coded data en_data_K corresponding to the K-th data block is stored in the server or a similar memory or storage device, the coded data en_data_K is compressed. Assuming that the compression step obtains the corresponding compressed coded data cp_en_data_K, when the data is restored, the compressed coded data cp_en_data_K is extracted according to the index of the coded data en_data_K and the original coded data en_data_K is decompressed. Note that this setting is an example given for the sake of simplicity of explanation. The actual operation must comply with the industry's operating standards. For example, the index can actually be stored by a separate index library, and data encoding and decoding and compression and decompression have corresponding standardization.

[0076] See also Figure 3 , the compression module 105 involves lossy compression and lossless compression. The data compression of the compression module 105 is a technology that compresses the storage space of a certain data or increases the amount of data storage in a certain space during data processing. The length of the record or block is usually shortened by eliminating data gaps, empty fields, redundant information and unnecessary data, with the aim of improving the utilization rate of computer storage space. Data compression is divided into lossless compression and lossy compression. Common lossless compression methods generally include: Huffman coding, arithmetic coding, run length coding and string table compression algorithm, etc. Common lossy compression methods generally include: predictive coding, transform coding and hybrid coding, etc. The compression module 105 recovers the space occupied by accommodating additional or redundant data through compression, which is equivalent to reaching a balance between the encoding module 104 and the compression module 105. The former causes space waste relative to ordinary storage, while the latter increases the amount of storage space.

[0077] See also Figure 4 In this embodiment, a method for implementing deduplication is achieved, including steps S1-S5.

[0078] See also Figure 4 ,Step S1, is to perform block processing on the file and calculate the fingerprint of each data block.

[0079] See also Figure 4 , Step S1, block processing typically includes: full file block algorithm, specified scale or fixed size block algorithm, content-based block algorithm, byte / bit level division, etc.

[0080] See also Figure 4, Step S1, the fingerprint calculation of the data block includes: traditional hash algorithm, in essence, the more brief fingerprint extraction method is to calculate the MD5 or SHA hash value of the text, but it is prone to the so-called avalanche effect, and the fingerprint calculated by MD5 or SHA will be different for extremely small and negligible text differences, because the probability of collision caused by such algorithms is extremely low. There are many other alternative fingerprint algorithms, such as K-shingle algorithm, Simhash algorithm and Minhash algorithm involving fingerprint deduplication, and the longest sentence signature algorithm is also one of the alternatives.

[0081] See also Figure 4 In step S2, the fingerprint of each data block is checked in the pre-stored fingerprint library to confirm whether the fingerprint of each data block already exists in the pre-stored fingerprint library 103.

[0082] See also Figure 4 In step S2, after the fingerprint of each data block is checked in the pre-stored fingerprint library 103, two different checking results will be generated, the answer is yes (ie, Y in step S3a) or the answer is no (ie, N in step S3b).

[0083] See also Figure 4 , step S3a, after the fingerprint of each data block is checked in the pre-stored fingerprint library 103, the check result is a flow direction of yes. The index of the data block needs to be saved, and the corresponding data block is abandoned. For example, after the fingerprint of the Kth data block is checked in the pre-stored fingerprint library 103, the check result is a flow direction of yes (Y), the index of the corresponding data block, such as the Kth data block, needs to be saved, and the corresponding data block, such as the Kth data block, is abandoned. The saved index provides a query basis for the data block for data operations such as tracing, recovery or restoration.

[0084] See also Figure 4 , step S3b, after the fingerprint of each data block is checked in the pre-stored fingerprint library 103, the flow direction in which the check result is negative is generated. The index of the data block needs to be saved, and the data block is subjected to a preset encoding process, and the encoded data is stored in the server. For example, after the fingerprint of the Kth data block is checked in the pre-stored fingerprint library 103, the flow direction in which the check result is negative (N) is generated, it is also necessary to save the index of the data block such as the Kth data block, and perform a preset encoding process on the data block such as the Kth data block, and store the encoded data en_data_K in the server or in a similar memory or storage device. Steps S3a and S3b are key steps for deduplication.

[0085] See also Figure 4Step S4 is the encoding process following step S3b. The encoding process includes, for example, encoding processing of error correction codes. When restoring data, the encoded data is extracted according to the index of the encoded data and its original data block is reconstructed according to the decoding principle of the error correction code. For example, the encoding process includes encoding processing of erasure codes (or error correction codes) such as cascaded low-density erasure codes, digital fountain codes, or Reed-Solomon codes.

[0086] See also Figure 4 , step S4, for example, the Kth data block is encoded into encoded data en_data_K, and when the data is restored, the encoded data en_data_K is extracted according to the index of the encoded data en_data_K, and the original data block of the encoded data en_data_K is reconstructed according to the error correction code such as the Reed-Solomon decoding principle.

[0087] See also Figure 4 , step S4 erasure coding can improve reliability, availability and storage efficiency. Reliability is measured in terms of fault tolerance, which is the number of concurrent failures that can occur continuously without losing data. Multiple identical copies are stored on different nodes and different sites by replication. With erasure coding, objects are encoded as data and parity fragments and distributed across multiple nodes and sites. This decentralized approach provides protection against both site and node failures. Compared to replication, erasure coding can improve reliability at a comparable storage cost.

[0088] See also Figure 4 ,Availability of Erasure Coding: Availability can be defined as the ability to retrieve an object when a storage node fails or is inaccessible. Compared to replication, erasure coding can improve availability at a comparable storage cost.

[0089] See also Figure 4 , Storage efficiency of erasure coding: For similar levels of availability and reliability, objects protected by erasure coding take up less disk space than the same objects protected by replication.

[0090] See also Figure 4, step S5 is the file reconstruction process following step S4. Repeat the reconstruction of all the encoded data until a series of original data blocks corresponding to all the encoded data are obtained, and they are restored and combined into the initial file. Finally, the original file is restored according to the segmented data blocks. For example, it is necessary to reconstruct the first data block, the second data block, the third data block, the Kth data block, the Nth data block, etc., and finally the 1-N segmented data blocks can be restored and combined into the initial file. It is worth noting that the illustrated step S5 only involves this related link when reading or restoring files. If the file is simply deleted and stored, step S5 can be omitted from the figure. Considering that the restoration of segmented fine-grained data blocks into files belongs to the scope of the prior art, it will not be described here. For example, a recovery module can be used to recombine the original data blocks and restore them into the initial file.

[0091] See also Figure 4 Step S5 can first decode the encoded data of step S4 to obtain the first data block, the second data block, the third data block, the Kth data block, and the Nth data block, and finally the 1st to Nth decoded segmented data blocks can be restored and combined into the original file.

[0092] See also Figure 5 , a step SX is inserted after step S4. Note that this step SX is optional, so in some embodiments, step SX can be omitted. and Figure 4 Compared with the embodiment, Figure 4 All technical features of the embodiments are applicable to Figure 5 However, Figure 5 compared to Figure 4 There are more steps SX. Figure 5 Examples and Figure 4 The repeated parts will not be described separately, and the specific details can be found in the above Figure 4 Embodiment of the invention.

[0093] See also Figure 5 Step SX is a step in which the coded data needs to be compressed. When the data is restored, the compressed coded data needs to be decompressed first, so step SX is to compress the coded data. For example, the Kth data block is encoded into coded data en_data_K, and the compression step SX can obtain the compressed coded data cp_en_data_K. When the data is restored, the compressed cp_en_data_K is decompressed to extract the coded data en_data_K.

[0094] See also Figure 5In step S5, the compressed data of step SX can be first decompressed to obtain the first data block, the second data block, the third data block, the Kth data block, and the Nth data block, and then the decompressed coded data can be decoded again. The first, second, Kth data block, and Nth data block can be obtained, and finally the 1st to Nth decoded segmented data blocks can be restored and combined into the original file.

[0095] See also Figure 4 , relates to a storage device for implementing deduplication, comprising: a data segmentation module, used to perform block processing on a file and obtain the segmented data blocks (step S1); a judgment module, used to extract the fingerprint of each data block and check the fingerprint of each data block in a pre-stored fingerprint library (step S2), so as to confirm whether the fingerprint of the data block exists in the fingerprint library; an encoding module, if the judgment module believes that the fingerprint of the data block is not in the fingerprint library, the encoding module performs a preset encoding process on the data block (step S3b); on the contrary, if the judgment module believes that the fingerprint of the data block is already in the fingerprint library, the encoding module does not perform the preset encoding process on the data block (step S3a), and abandons the corresponding data block. Note that this embodiment is an optional and non-essential example.

[0096] See also Figure 4 The encoding process (step S4) performed by the encoding module includes the encoding process of the error correction code and when restoring the data, for example, in step S5, the encoded data can be extracted and its original data block can be reconstructed according to the decoding principle of the error correction code; all the encoded data are reconstructed repeatedly until the original data blocks corresponding to all the encoded data are obtained; the recovery module of the storage device (recovering the data blocks into files) reassembles the original data blocks, for example, the first, second to Nth data blocks, into the original file. Note that this embodiment is an optional and non-essential example.

[0097] See also Figure 5 In an optional but not essential embodiment, data compression may be performed before the encoded data is stored in the server, such as in step SX, and when the data is restored, such as in step S5, the compressed encoded data is extracted and the original encoded data is decompressed. Figure 5 The example is compared with Figure 4 It is to insert SX between S4 and S5.

[0098] See also Figure 2Compared with centralized storage, a distributed storage system is a storage system that stores data in multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. The storage server becomes the bottleneck of system performance and the focus of reliability and security, and cannot meet the needs of large-scale storage applications. Distributed network storage systems use an extensible system structure, use multiple storage servers to share storage loads, and use location servers to locate storage information. It not only improves the reliability, availability, and access efficiency of the system, but is also easy to expand.

[0099] See also Figure 2 The storage device for implementing deduplication shown in the figure is suitable for distributed storage or centralized storage, but in a distributed storage system, the storage device for implementing deduplication shown in the figure has better reliability and security characteristics, and uses multiple storage servers to share the storage load (for example, the encoded data is stored in multiple distributed servers). It is difficult to extract the information required for decoding from multiple distributed servers without trusted conditions. Obviously, it not only improves the credibility, intrusion resistance and pressure of centralized data storage, but also has error correction capabilities based on deduplication conditions.

[0100] See also Figure 3 In an application example, the present application takes the storage device as a storage array and the host as a master control server as an example to illustrate the specific implementation process of the technical solution: when the master control server sends business data, it first establishes a sample database on the master control server according to the present application. In addition, the master control server needs to communicate with the storage array through industry standard protocols (such as SCSI / SATA / SAS / FC) or custom protocols, and the master control server executes storage commands or other data operation commands to send the data in the sample database to each array. After each storage array receives a comparison data request or similar operation from the master control server, it compares the received data with the data stored in the array and performs other processing. After the above-mentioned specific implementation, the duplicate data of each storage array will be processed and deleted at the same time, and it will not affect other normal business data processing between the master control server and the storage array. The storage array can be replaced with a hard disk such as an SSD disk or a mechanical disk, and the solution is also applicable to the embodiment in which the storage device is a hard disk.

[0101] See also Figure 2, perform block processing on the file, calculate the fingerprint of each data block, and use a very small data to mark the unique information of a certain file. There are many algorithms (such as MD5 algorithm and SHA algorithm) that can quickly obtain the corresponding unique fingerprint value. The SHA algorithm is an irreversible data encryption algorithm that can only calculate the fingerprint but cannot infer the content through the fingerprint. The SHA algorithm can convert a data that is less than 2^64 into a 160-bit non-repeating fingerprint. The key point is that the SHA algorithm is very fast. Therefore, to compare whether two data are the same, you can calculate the corresponding fingerprint and then compare the fingerprints without byte-by-byte proofreading of the data. It is much more efficient.

[0102] See also Figure 2 According to the SHA256 algorithm, the probability of two data fingerprints being repeated in 4.8*10^29 data is less than 10^(-18.10)^(-18). In fact, the calculation results show that the reliability is almost 100%. Therefore, it is almost impossible for two different fingerprints to be repeated.

[0103] See also Figure 2 In an optional but not essential embodiment, during the block processing stage, a specified scale is set and the file to be processed is segmented according to the specified scale to obtain a series of data blocks, and the fingerprint of each data block is calculated based on the specified scale. This design is very effective when there is less similar data, for example, any two data have a large difference, and there is no complex block segmentation algorithm, so it saves CPU resources.

[0104] See also Figure 2 In an optional but not essential embodiment, during the block processing stage, the file to be processed is divided into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale. This design is extremely effective when there is a lot of similar data, for example, any two data have small differences. When there is a lot of similar data, it is a more difficult deduplication problem. For example, if a file is simply added with a character or the value of a very small number of bits is changed or a data is deleted, and then rechecked, the file cannot be found after being sliced ​​with a fixed length and specified scale, resulting in the inability to duplicate the deleted data.

[0105] See also Figure 2For example, there is a file file_A. Simply adding a character or changing the value of a few bits or deleting a data to file_A changes it to file_B. Assume that file_A includes ab000…111 and at the same time, assume that a character is added to file_A. Assume that file_A includes abc000…111. After slicing with a fixed-length specified scale, it is impossible to find the same data block as before, making deduplication difficult. Calculating data block fingerprints based on elastic scales slightly increases CPU memory consumption and basically does not affect the real-time data processing efficiency, because storage mainly involves processing the host's I / O read and write responses. At this time, the capacity of variable-length deduplication and fixed-length deduplication is ten to three. In application scenarios such as image formats, variable-length deduplication becomes extremely efficient.

[0106] See also Figure 6 , known Figure 1 The multiple copies of data D1 are duplicate data, the multiple copies of data D2 are duplicate data, and the remaining multiple copies of data D3 are duplicate data. The previous article introduced that without establishing duplicate data filtering rules, the result of deduplication of multiple copies of data D1 is as shown in the figure. The duplicate data covered by the shaded part in the previous figure is deleted, and the same is true for multiple copies of data D2-D3.

[0107] See also Figure 6 The total amount of deduplicated data D1-D3 is recorded as D-DATA, which is essentially the necessary data retained after the duplicate data is deleted. Compared with the previous R-DATA and the current D-DATA, the total capacity of the storage system can be increased by approximately 60-70%, and the current capacity loss rate S-LOSS is greatly reduced, recovering more than 50%.

[0108] See also Figure 2 , for example, transistors or magnetic elements may be used in the carrier of data or information. However, the carrier structure may not only produce unexpected physical structural errors in the process preparation stage, but also produce hidden errors in the later data operation stages such as data transmission, data reading or data storage. Cracking or dirt adhesion caused by collision or falling of the carrier structure during transportation and use are also structural errors. Structural errors are hard errors, and the external phenomenon of hard errors is that the structure at one or several positions of the carrier material may repeatedly have errors. Hidden errors are soft errors, and the external phenomenon of soft errors is that random errors may occur in the carrier structure at any position. Hard errors can improve the yield of the generated carrier structure by improving the production process of preparing the carrier material or improving the material, while soft errors can only obtain high data reliability through error correction schemes such as erasure code technology.

[0109] See also Figure 6, the deletion of duplicate data D1-D3 is Figure 2 As mentioned above, for all-flash storage with high response requirements, fixed-length deduplication can be used, which is fast. For storage such as archiving and backup, variable-length deduplication can be used with a high deduplication rate to save costs. Furthermore, the combination of deduplication and compression will be more meaningful for the substantial release of storage space and the improvement of the useful area of ​​storage space. However, the problem is that regardless of the data errors in compression or the data errors in encoding processing, the error correction code or erasure code encoding processing superimposed on compression will exponentially amplify the error rate of the reconstructed original data block. Based on the explanation, take Reed-Solomon as an example. Reed-Solomon depends on the data block itself and on redundant data such as the unit matrix. If compression changes the original data form of the unit matrix, or compression changes the data block itself, error correction is the result of a large number of matrix operations on the unit matrix and the original data block. Obviously, the total error of the reconstructed original data block is not only related to the compression error, nor is it only related to the encoding error, but it presents an exponential amplification of the error, because any erroneous data point will cause a large area of ​​​​error in the calculated matrix. The hidden and difficult to detect problem is that Figure 6 The deletion results of duplicate data D1-D3 are indeed shown, but the apparent deduplication results may have the aforementioned exponentially amplified errors. Data errors have many sources, such as the aforementioned hard errors or soft errors, which are potential factors. In addition, there are many other reasons for data errors during transmission or storage, such as naturally existing noise or interference, parasitic coupling transmission between adjacent data lines, etc.

[0110] See also Figure 2To solve the above problems, an optional method is to divide the to-be-processed file into a series of data blocks of different scales during the block processing stage, and calculate the data block fingerprint based on the elastic scale. The larger the elastic scale when the series of data blocks are divided into blocks, the smaller the data ratio of the check block used in the encoding process relative to the data to be encoded. Or the smaller the elastic scale when the series of data blocks are divided into blocks, the larger the data ratio of the check block used in the encoding process relative to the data to be encoded. For example, the preset encoding process is implemented on the data block, taking the error correction code as an example. Under this mode and through the dual processing of encoding and compression, the error rate of the unit matrix and the Vandermonde matrix is ​​greatly reduced. For flash memory, the storage space released by only deduplication or only compression may only account for 3 / 10 of the storage space released by their cooperation. Although the encoding process of error correction code or erasure code superimposed on compression will exponentially amplify the error rate of the reconstructed original data block, in the above mode, the possible potential error points of a large number of matrix operations of the unit matrix and the original data block are discrete, reducing the failure probability that restricts the encoding reliability of error correction code or erasure code. The discretization of a single error point of a hard error or soft error in the unit matrix and the Vandermonde matrix may be able to recover a corresponding entire row of data in the matrix, which is equivalent to retaining more original information in the remaining residual matrix when discarding the erroneous data. Then the success rate of restoring the original data block by the residual matrix will be greatly improved. The integrity of information can also be maintained on noisy media (such as radio waves, communication lines, etc.), and all transmitted information can be correctly reconstructed after storage or reception, even if there are some errors in the stored or received information. The aforementioned data to be encoded includes, for example, data blocks.

[0111] See also Figure 2In an optional embodiment, the larger the elastic scale when the data block is divided into blocks, the smaller the data ratio of the check block used in the encoding process relative to the data to be encoded. The smaller the elastic scale when the data block is divided into blocks, the larger the data ratio of the check block used in the encoding process relative to the data to be encoded. They have the same meaning. A more reasonable solution is that if the data ratio of the check block used in the encoding process of any data block relative to the data to be encoded is not within a predetermined range, it is necessary to call back the actual value of the elastic scale used when any data block is cut into blocks until the data ratio of the check block used in the encoding process of any data block relative to the data to be encoded is adjusted to within the predetermined range. The data ratio is peak-shaving and valley-filling, which is suitable for forcibly dispersing possible error points involved in a large number of matrix operations. The callback of the elastic scale usually allows fine-tuning of the scale size of the cut data blocks when the data blocks are cut into blocks. Sometimes this is not the case. Take the sliding window-based block method as an example. At this time, the callback elastic scale does not simply adjust the size of the data block scale to make it fine-tuned. For example, the Rabin algorithm determines the data block boundary, which should satisfy that for a sequence S=s1, s2, ...sn, if and only if the edge of the window stops at a certain k position, there is a subsequence W=s(k-l+1), s(k-l+2), ...sk, and the fingerprint function calculation result of the subsequence is h(W)mod D=r, then there is a D-match at the k position. When calling back the elastic scale, for example, the boundary of the segmented data block should be satisfied to be at the k position (meeting h(W)mod D=r).

[0112] See also Figure 2In the optional example, taking RS coding as an example, the change trend of the data ratio of the check block used in the coding process relative to the data to be coded is a reverse dynamic change with the change of the elastic scale when the data block is divided. For example, the larger the elastic scale, the smaller the data ratio of the check block relative to the data to be coded, and the smaller the elastic scale, the larger the data ratio of the check block relative to the data to be coded. If the size of the elastic scale used when the data block is cut is called back, one solution is to set a backup of the file, and when the elastic scale is called back, the backup file can be re-divided until the elastic scale used for each data block is adjusted to be within a predetermined range. To call back the size of the elastic scale used when the data block is cut, an optional solution is to re-combine the series of data blocks obtained from the previous division into a file, and when the elastic scale is called back, the combined file at this time can be re-divided, and the elastic scale used for each data block is adjusted in a cycle until the elastic scale used for each data block is adjusted to be within a predetermined range. By cyclically executing the division, re-combination and re-division of the data block and thereby adjusting the elastic scale corresponding to each data block, a large number of potential error points involved in matrix operations can be discretized. This prevents errors from being concentrated in a few data blocks or a limited number of data blocks and their checksum redundant data, thus ultimately reducing storage and compression errors.

[0113] See also Figure 3In the optional example, the file is processed by block processing, and the fingerprint of each data block is calculated; in the block processing stage, the file to be processed is divided into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale. The fingerprint of each data block is checked in the pre-stored fingerprint library to confirm whether it exists: if it is, the index of the data block is saved and the corresponding data block is abandoned; if it is not, the index of the data block is saved, and the data block is subjected to preset encoding processing, and the encoded data is stored in the server. The encoding processing includes error correction code encoding processing, and the encoded data is compressed before being stored in the server. The change trend of the data ratio of the check block used in the encoding processing relative to the encoded data is set to change dynamically in the opposite direction with the change of the elastic scale when the data block is divided into blocks. In an optional example, a further solution is that if the data ratio of the check block used for encoding processing of any data block relative to the data to be encoded is not within a predetermined range, the actual value of the elastic scale used for the block segmentation of any data block is called back again until the data ratio of the check block used for encoding processing of any data block relative to the data to be encoded is adjusted to within the predetermined range. A further solution is to call back the elastic scale used for the block segmentation of any data block: wherein a series of data blocks obtained by the last segmentation before the call back is reassembled into an intermediate file, and when the elastic scale is called back, the next segmentation needs to be re-implemented on the intermediate file at this time, and the elastic scale used for each data block is adjusted in a cycle until the elastic scale used for each data block is adjusted to be within the predetermined range. In this optional but not necessary embodiment, the segmentation (last segmentation) reassembly (intermediate file) and re-segmentation (next segmentation) of the data block are executed in a cycle, and the elastic scale corresponding to each data block is adjusted thereby. A larger number of loops can better disperse erroneous data, but it will also consume more CPU resources or GPU resources. Therefore, one way to achieve a compromise is to adjust the predetermined range of the data ratio of the check block used in the constraint coding processing relative to the encoded data to a reasonable range, so as to achieve a better effect of eliminating or reducing errors without excessively consuming hardware resources.

[0114] See also Figure 3In an optional example, the data segmentation module is used to perform block processing on the file to obtain the segmented data blocks. In the block processing stage, the to-be-processed file is segmented into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale. If the judgment module believes that the fingerprint of the data block is not in the fingerprint library, the encoding module performs a preset encoding process on the data block; on the contrary, if the judgment module believes that the fingerprint of the data block is already in the fingerprint library, the encoding module does not perform a preset encoding process on the data block, and the corresponding data block is abandoned. The encoding process implemented by the encoding module includes error correction code encoding, and the encoded data is compressed before being saved to the storage device. The change trend of the data ratio of the check block used in the encoding process relative to the encoded data is set to change dynamically in the opposite direction with the change of the elastic scale when the data block is segmented. Further measures, such as the scheme of re-calling back the actual value of the elastic scale used by any data block when it is segmented, are not described separately. In addition, the scheme of cyclically adjusting the elastic scale used by each data block is not described separately, and the relevant description in the previous text can be referred to.

[0115] The above description and drawings provide typical embodiments of specific structures of specific embodiments, and the above invention provides existing preferred embodiments, but these contents are not intended to be limiting. For those skilled in the art, various changes and modifications will undoubtedly be obvious after reading the above description. Therefore, the attached claims should be regarded as covering all changes and modifications to the true intent and scope of the present invention. Any and all equivalent scopes and contents within the scope of the claims should be considered to still be within the intent and scope of the present invention.

Claims

1. A method for implementing deduplication, characterized in that: include: Perform block processing on the file and calculate the fingerprint of each data block; Check the fingerprint of each data block in the pre-stored fingerprint library to confirm whether it exists: If yes, the index of the data block is saved and the corresponding data block is discarded; If not, the index of the data block is saved, and a preset encoding process is performed on the data block, and the encoded data is stored in the server; In the block processing stage, the file to be processed is divided into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale. The larger the elastic scale when the series of data blocks are divided, the smaller the data ratio of the check block used in the encoding process relative to the data to be encoded, and the smaller the elastic scale when the data blocks are divided, the larger the data ratio of the check block used in the encoding process relative to the data to be encoded, so as to set the change trend of the data ratio of the check block used in the encoding process relative to the data to be encoded to change dynamically in the opposite direction with the change of the elastic scale when the data blocks are divided; Among them, if the data ratio of the check block used for the encoding processing of any data block relative to the data to be encoded is not within a predetermined range, it is necessary to readjust the actual value of the elastic scale used for the slicing of any data block until the data ratio of the check block used for the encoding processing of any data block relative to the data to be encoded is adjusted to within the predetermined range, and the data ratio is suitable for forcibly dispersing the error points involved in the matrix operation of the original data block.

2. The method according to claim 1, characterized in that: When checking the fingerprint of a data block in the fingerprint library, a HASH search method is used to search whether the fingerprint of the data block exists in the fingerprint library.

3. The method according to claim 1, characterized in that: The encoding process includes error correction code encoding process. When restoring data, the encoded data is extracted according to the index of the encoded data and its original data block is reconstructed according to the decoding principle of the error correction code.

4. The method according to claim 3, characterized in that: Repeatedly reconstruct all the encoded data until a series of original data blocks corresponding to all the encoded data are obtained, and then they are restored and combined into the original file.

5. The method according to claim 1, characterized in that: The encoded data is compressed before being stored in the server. When the data is restored, the compressed encoded data is extracted according to the index of the encoded data and the original encoded data is decompressed.

6. The method according to claim 5, characterized in that: The compression includes lossless compression using at least one of Huffman coding, arithmetic coding, run length coding or string table compression algorithms.

7. A storage device for implementing deduplication, characterized in that: include: A data segmentation module is used to perform block processing on the file to obtain segmented data blocks; A judgment module is used to extract the fingerprint of each data block and check the fingerprint of each data block in a pre-stored fingerprint library to confirm whether the fingerprint of the data block exists in the fingerprint library; The encoding module performs a preset encoding process on the data block if the judgment module determines that the fingerprint of the data block is not in the fingerprint library; On the contrary, if the judgment module believes that the fingerprint of the data block is already in the fingerprint library, the encoding module does not perform the preset encoding process on the data block and abandons the corresponding data block; In the block processing stage, the file to be processed is divided into a series of data blocks of different scales, and the fingerprint of each data block is calculated based on the elastic scale. The larger the elastic scale when the series of data blocks are divided, the smaller the data ratio of the check block used in the encoding process relative to the data to be encoded, and the smaller the elastic scale when the data blocks are divided, the larger the data ratio of the check block used in the encoding process relative to the data to be encoded, so as to set the change trend of the data ratio of the check block used in the encoding process relative to the data to be encoded to change dynamically in the opposite direction with the change of the elastic scale when the data blocks are divided; Among them, if the data ratio of the check block used for the encoding processing of any data block relative to the data to be encoded is not within a predetermined range, it is necessary to readjust the actual value of the elastic scale used for the slicing of any data block until the data ratio of the check block used for the encoding processing of any data block relative to the data to be encoded is adjusted to within the predetermined range, and the data ratio is suitable for forcibly dispersing the error points involved in the matrix operation of the original data block.

8. The storage device for implementing deduplication according to claim 7, characterized in that: The encoding process performed by the encoding module includes an error correction code encoding process. When restoring data, the encoded data is extracted and its original data block is reconstructed according to the decoding principle of the error correction code; as well as Repeatedly reconstructing all the coded data until a series of original data blocks corresponding to all the coded data are obtained; A recovery module of the storage device reassembles the original data blocks into the original file.

9. The storage device for implementing deduplication according to claim 7, characterized in that: The encoded data is compressed before being stored in the storage device. When the data is restored, the compressed encoded data is extracted and the original encoded data is decompressed.

10. The storage device for implementing deduplication according to claim 9, characterized in that: Data compression includes lossless compression using any of Huffman coding, arithmetic coding, run-length encoding, or string table compression algorithms.

Images