Hashing method and system for deduplicating data by skipping selected data

Abstract

The present invention discloses a system for calculating fingerprints across data sets by the following method, the method comprising: identifying a data set to be hashed, the data set comprising a data block set; identifying data to be skipped within the data set; generating a hash by a hash engine for each data block in the data block set except for the data to be skipped; and compressing the data.

Description

Technical Field

[0001] The embodiments discussed herein relate to hashing, deduplication, and compression. Background Art

[0002] A dataset may include duplicate data. Removing or reducing the amount of duplicate data from a dataset (commonly known as deduplication) can reduce storage space and achieve greater compression of the dataset. A hashing engine can be used to identify duplicate data within a dataset.

[0003] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is provided merely to illustrate one exemplary technology area where some embodiments described herein may be practiced. Summary of the Invention

[0004] A system for computing fingerprints across a data set by the following method, the method comprising: identifying a data set to be hashed, the data set comprising a set of data blocks; identifying data to be skipped within the data set; generating, by a hash engine, a hash for each data block in the set of data blocks within the data set except for the data to be skipped; and compressing the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Illustrative embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0006] Figure 1 An exemplary data deduplication system is shown;

[0007] Figure 2 shows a data set that can be hashed at a hash block size and compressed at a compression block size;

[0008] Figure 3 Another exemplary data deduplication system having multiple hash engines is shown;

[0009] Figure 4 shows a data set that can be hashed by different hash engines and compressed at compression block sizes, wherein each hash engine hashes at a different hash block size;

[0010] Figure 5 A flowchart illustrating an exemplary method for hashing a data set is shown;

[0011] Figure 6 A flow chart illustrating another exemplary method for hashing a data set is shown;

[0012] Figure 7 A flow chart illustrating yet another exemplary method for hashing a data set;

[0013] Figure 8 A diagrammatic representation of a machine is shown in exemplary form as a computing device within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. DETAILED DESCRIPTION

[0014] A hash engine can be used to identify duplicate data within a dataset. Data deduplication is a key component of data reduction. Using a hashing algorithm, a hashing engine can generate a mapping of any size dataset to a fixed-size set of values (called a hash). A hash is a unique set of bits of a specific size with a low probability of duplication. Running a dataset through a hashing algorithm can produce an output of a much smaller data bit size. For example, running 32k bits of data through a hashing algorithm can result in an output as small as 8 bits, 16 bits, 32 bits, and so on.

[0015] Hashing can be used to efficiently identify potential duplicate data within a dataset that can be deduplicated. A hashing algorithm takes a dataset and generates a unique set of bit-sized results. Hashes across one or more datasets are compared to find potential exact matches that can be deduplicated. Multiple datasets can also be run through the hashing algorithm. Because the hashing engine can process multiple datasets, the results can be compared. A hash match between multiple datasets can indicate that the data corresponding to the matching hash is likely a duplicate. All but one instance of the matching data can be deleted from the multiple datasets and replaced with a pointer to the remaining instance of the matching data. Deduplication reduces the storage space required to store the multiple datasets. If the data is compressed, duplicate data will not be compressed, resulting in greater compression.

[0016] Traditionally, when hardware is used to compress a dataset, the dataset is compressed at the same dataset size as the hash size. In other words, if a dataset is compressed to 32k bits, the data is hashed to 32k bits. Coupling the compression size and hash size can result in trade-offs in storage size, efficiency, and the number of duplicates. For example, a larger compression size results in better data compression across the dataset. However, as the hash size increases, the number of matching hashes decreases, which reduces the efficiency of deduplication. Similarly, as the hash size decreases, the number of duplicates increases, but the resulting compression decreases due to lower efficiency for the same size. Therefore, traditionally, coupling the compression size and hash size results in a tension between these two characteristics. Typically, neither the compression size nor the hash size is optimal; instead, a midpoint is chosen that results in suboptimal compression and deduplication.

[0017] Various aspects of the present disclosure address these and other shortcomings of conventional compression and deduplication techniques. Described herein are systems and methods for improving both compression and deduplication by decoupling the compressed size and the hash size. By separating the hash size from the compressed size, the system is allowed to make the hash size much smaller, and the smaller the hash, the more efficient the hash is at finding duplicates. For example, when hashing at 32k bits, the likelihood of getting multiple 32k bits across a data set is much lower than when hashing at 1k bits or 512 bits. Some of the systems described herein provide a mechanism to specify a hash size of any size, regardless of the compressed size of the data set.

[0018] Some of the benefits provided by the present disclosure include enhanced compression and data deduplication. For example, by separating the hash size and the compressed size, very large data sizes can be used to enhance compression, while a small hash size, on the same data set, increases the amount of hashing on the data set, thereby improving "hash hits" or the ability to identify and remove duplicate data across the data set.

[0019] In one example, when a first data set A is one megabyte in size and the hash size is much smaller (e.g., 1 kb), within that data set, there may be a block of data with a 1 kb hash that is duplicated elsewhere in that data set or in another data set B. In either case, the duplicate data may be compressed once, and for the other duplicate data, a pointer to the compressed data in the first data set A is tracked so that if the duplicate data is accessed with respect to data set B, the duplicate data can be retrieved from the first data set A. In at least one embodiment, the pointers / locations are identified and managed by a primary storage control system, which may be local or remote to the hash engine.

[0020] Various aspects of the present disclosure may also identify data to skip during hashing. The benefits of this technique may increase the number of hash hits and the overall efficiency of hashing and compression.

[0021] Some aspects of the present disclosure may utilize multiple hash engines, each of which may perform different hash operations on the same data set in a single pass. This technique may further improve hash hit rates and compression rates. For example, the multiple hash engines may have the following properties: running different hash algorithms, running with different hash block sizes, running with different starting offsets, running with different skip capabilities / settings, and / or any combination of the above.

[0022] Go to the attached figure, Figure 1An exemplary data deduplication system 100 is shown. The data deduplication system 100 can be arranged according to at least one embodiment described herein. The exemplary data deduplication system 100 can include one or more data sources 105, a data deduplication manager 110, one or more hash engines 115, a data storage device 120, and a compression manager 125. In at least one embodiment, the data deduplication manager 110, the hash engine 115, and the compression manager 125 can be integrated into an accelerator card 155.

[0023] Data source 105 may include any type of device, such as a computing device, a client device, a data storage device, etc. Data source 105 may receive data and send the data to deduplication manager 110 for hashing. Data source 105 may include an interface for providing parameters to deduplication manager 110, through which the data may be deduplicated.

[0024] Deduplication manager 110 may include a central processing unit (CPU) or be part of a CPU, or may be a module executable by the CPU. Additionally or alternatively, deduplication manager 110 may be a separate unit that communicates with the CPU. For example, the CPU may offload various tasks, such as hashing, encryption, and compression, to deduplication manager 110.

[0025] Data source 105 can be configured to communicate with deduplication manager 110, for example, via a bus or network. Data can be sent from data source 105 to deduplication manager 110, requesting that an operation be performed on the data, such as hashing, encryption, and / or compression. The request can include various parameters and metadata associated with the data. Regarding the execution of the hashing operation, deduplication manager 110 is communicatively coupled to hash engine 115.

[0026] The hash engine 115 may include circuitry, software, or a combination of circuitry and software configured to perform a hash algorithm on a data set, such as data received from the data source 105. The hash engine 115 may be configured to perform any hash algorithm, such as MD5, SHA-1, SHA-2 (224, 256, 384, 512), or any other hash algorithm, or for error detection (e.g., checksum, cyclic redundancy check (CRC)).

[0027] The data storage device 120 may include any memory or data storage device. In some embodiments, the data storage device 120 may include a computer-readable storage medium for carrying or having computer-executable instructions or data structures stored thereon. The computer-readable storage medium may include any available medium that can be accessed by a general-purpose or special-purpose computer (e.g., a processor). For example, the data storage device 120 may include a computer-readable storage medium, which may be a tangible or non-temporary computer-readable storage medium, including a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage device, a magnetic disk storage device or other magnetic storage device, a flash memory device (e.g., a solid-state storage device), a non-volatile memory, or any other storage medium that can be used to carry or store the required program code in the form of a computer-executable instruction or data structure and can be accessed by a general-purpose or special-purpose computer. The combination of the above may be included in the data storage device 120. In the depicted embodiment, the data storage device 120 is separated from the accelerator card 155. In some embodiments, the data storage device 120 may be part of an accelerator card 155, which may interface with the host using any type of interface, such as an eight-lane PCI Express 3.0 interface. In at least one embodiment, the data storage device 120 may include multiple distributed data storage devices.

[0028] The compression manager 125 may include circuitry, software, or a combination of circuitry and software configured to perform a compression algorithm on a data set (e.g., data received from the data source 105 and / or data that has been hashed). The compressed data 150 may be stored in the data storage device 120. The compression manager 125 may be configured to perform any compression algorithm, such as gzip / zlib / Deflate L4, eLZS / LZS, or any other compression algorithm.

[0029] The deduplication manager 110 may include a slice manager 130 and a skip manager 135. The slice manager 130 may include a hardware module, a software module, or a combination of hardware and software configured to manage hash operations at a hash size that is smaller than the size of the compressed data set. The skip manager 135 may include a hardware module, a software module, or a combination of hardware and software configured to identify some data in the data set that can be skipped during hashing.

[0030] In operation, the deduplication manager 110 may receive a data set from the data source 105. Traditionally, the hash block size is the same as the compressed data set size. The slice manager 130 decouples the hash block size from the compressed data set size and allows the hash block size used for deduplication to be different from the compressed data set size. The data set may include or be organized based on data block sets.

[0031] Slice manager 130 can identify a data set to be deduplicated, such as a data set from data source 105. Slice manager 130 can select a hash block size that is smaller than the compression block size (later used by compression manager 125). Slice manager 130 can define a set of data blocks within the data set based on the hash block size and can instruct hash engine 115 to generate a hash for each data block in the set of data blocks within the data set. Hash engine 115 can generate a hash for each data block in the set of data blocks within the data set, which hash can be stored in hash index 140. Deduplication manager 110 can dedupe data blocks within the data set based on their respective hashes. For example, to dedupe data blocks, deduplication manager 110 can delete duplicate data blocks and associate a pointer to the deleted data blocks with another location that also stores the same data as the duplicate. This pointer can be stored as pointer data 145. In at least one embodiment, compression manager 125 can compress the data set based on the compression block size. In at least one embodiment, compression manager 125 does not compress duplicate data or pointers to duplicate data. In at least one embodiment, hashing and compression are performed on a data set in a single pass. Additionally or alternatively, deduplication manager 110 can be directed to perform various operations and provide the results to other systems to perform the actual deduplication and manage and / or generate pointers and / or one or more tables.

[0032] The hash block size can be any size. In at least one embodiment, the hash block size can be between 1k and 4k bits. In at least one embodiment, when deduplicating structured or tabular data (e.g., database-type structures), the data block size can be relatively small, so a smaller hash block size can be used. For example, for unstructured data, a larger hash block size can be used.

[0033] In at least one embodiment, the hash block size is a divisor of the total data set such that each hash block is the same size. For example, for a total data set of 16k bits, the hash block size can be a divisor of 16k, such as 8k, 4k, 2k, 1k, etc. The hash block size can be a whole number, an integer, etc. In another embodiment, the hash block size is not a divisor of the total data set. In this example, all hashes except the last hash are retained, and the last hash can be used or discarded. For example, when the last bit is only 1k, but the hash block size is 4k bits, the last hash of 1k bits can be ignored or discarded. In this example, the data associated with the hash that is ultimately ignored or discarded may not be a candidate for deduplication.

[0034] In at least one embodiment, the deduplication manager 110 may determine that some data within a data set can be skipped for hashing. In at least one embodiment, the deduplication manager 110 receives an instruction from the data source 105 to skip some data within the data set. Additionally or alternatively, the deduplication manager 110 may use a skip manager 135 to automatically analyze the data set to determine whether there is data in the data set to be skipped. The skip manager 135 may, for example, identify certain types of data that can be skipped, such as header data, footer data, database fields, predetermined data types, data protection fields (e.g., NVMe protection data, data integrity field (DIF), data integrity extension (DIX), T10-DIF or T10-DIX, etc.). In one example, a company-wide email is sent along with a calendar invitation to a company party. Each email will include calendar data that is the same for all company employees. Each email will also include a header, which will be slightly different for each employee. In a traditional system, the email would not be deduplicated due to the different headers. The skip manager 135 may identify the header as a skip candidate so that the header is skipped during the deduplication process and the calendar data may still be deduplicated.

[0035] In at least one embodiment, the data from the data source 105 includes instructions or metadata about the data, which the skip manager 135 can use to determine data to skip during deduplication. For example, the data from the data source 105 can include information and / or metadata about the data from the data source, such as the number of bits of data, header data, footer data, header size, footer size, index, index header, etc. The information and / or metadata about the data from the data source can also include an indication of a repeating pattern in the data. For example, the data provider (e.g., the data source 105) can indicate a header pattern in the data, which can include an offset, a pattern size, and a skip count. The skip manager 135 can instruct the hash engine 115 to skip data of a certain size starting from an offset, how to identify the next data to skip, and a count of the total number of skips to perform. In this way, the hash engine 115 can run a hash on all data that has not been skipped, which can achieve a greater number of hash hits. Exemplary patterns may include one or more of pattern description, skipping "x" digits, processing "y" digits, skipping "z" digits, processing "a" digits, repeating, etc., where x, y, z, and a may be any numbers including numbers that are the same as each other, or some or all of them may be different from each other.

[0036] Similar techniques can be used for tabular or database type data or any other data that has an index header and data. For such data, the index header is skipped and the data can be hashed, deduplicated and / or compressed. In at least one embodiment, the hash block size can be smaller than the amount of data between skips. In at least one embodiment, the hash block size can be larger than the amount of data between skips so that the data on both sides of the "skip" can be hashed together. In at least one embodiment, the "unskipped" data is compressed and the "skipped" data is not compressed. In at least one embodiment, the skip capability is a layer where one set of skip rules is applied first, then a second set of skip rules can be applied to the remaining unskipped data, and so on.

[0037] In at least one embodiment, the skip manager 135 can automatically identify skip candidates. For example, the skip manager 135 can analyze the data before hashing to identify skip candidates, which may include data similar to "skippable" data. The skip manager 135 may have a set of criteria to use when analyzing the data. This set of criteria can identify various types of data that can be skipped, such as headers, footers, etc. When the skip manager 135 identifies these types of data as skippable, it can add a flag to the data so that the hash engine 115 and / or compression manager 125 can skip that specific data when hashing / compressing the data. In at least one embodiment, training data can be sent to the skip manager 135, which can be used to create an artificial intelligence (AI) unit to process the data to find "skippable" sections. In at least one embodiment, the AI unit can be supervised or unsupervised. In at least one embodiment, the skip manager 135 can use input received from a system administrator to refine the AI unit. For example, the skip manager 135 can identify "skip candidates" that can be presented to the system administrator via an interface. The system administrator may provide input as to whether these skip candidates should be skipped. The skip manager 135 may use the system administrator data to refine the AI unit to better identify skip candidates.

[0038] In at least one embodiment, the data includes metadata that identifies skippable data. The skippable data may include predetermined criteria for precise or predetermined formatted data. For example, the predetermined criteria may include NVMe protected data, such as T10-DIF data, T10-DIX data, etc.

[0039] By identifying NVMe protection fields, the skip manager 135 and hash engine 115 can avoid hashing those NVMe protection fields that may include DIF data, metadata, etc. Similarly, the compression manager 125 can avoid compressing those NVMe protection fields.

[0040] In at least one embodiment, when the skip manager 135 encounters NVMe protection fields within a data set, the skip manager 135 can provide precise instructions to the hash engine 115 to skip those NVMe protection fields.

[0041] In at least one embodiment, the skip manager 135 may determine that the dataset is formatted using "NVMe protection, DIF configuration". The skip manager 135 may determine that under this configuration, the DIF data, the metadata, or both the DIF data and the metadata may be skipped. The metadata associated with the dataset may indicate which of these selections to make. If not, the skip manager 135 may determine which to make or may rely on a default setting for the selection. The skip manager 135 may instruct the hash engine 115 to hash a particular portion of the dataset at a given data hash size, then skip the metadata and / or DIF data, and then hash the next portion of the dataset at the given data hash size, until all data within the dataset (except the skipped data) has been hashed.

[0042] These techniques for identifying and skipping hashed, deduplicated, and compressed portions of a data set can provide significant benefits to the CPU. In some embodiments, by performing this skipping in hardware, software CPU time is reduced. For database structures, these techniques can achieve deduplication rates of 2x, 3x, or even 5x.

[0043] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to the deduplication system 100. All of the constructs provided herein may be implemented in a stateless operation, where all data to be operated on is provided in a single command. Additionally or alternatively, for a stateful structure where data that was part of the single command continues to be submitted in a subsequent command, these functions may continue where they left off. Furthermore, all of the constructs provided herein may be implemented in one or more stateful operations.

[0044] Figure 2 A data set 200 is shown that can be hashed with a hash block size 210 and compressed with a compressed block size 220. The hash block size 210 can be a different value than the compressed block size 220. As shown, the compressed block size 220 appears to be equal in size to the data set 200, but the compressed block size 220 can be any size, including a size that is smaller than the size of the data set 200. The hash block size 210 can be any size, including a size that is larger than the compressed block size.

[0045] Figure 3 Another exemplary data deduplication system 300 is shown having multiple hash engines 315a, 315b, 315n. The data deduplication system 300 may be similar to Figure 1 The deduplication system 300 may include multiple hash engines 315a, 315b, 315n, which may be similar to the deduplication system 100. Figure 1The deduplication system 300 can be used to deduplicate any type of data, including multiple similar documents, such as multiple versions of a single document. For example, if a user generates a text-based document and then makes a new version of the text-based document but changes spaces, additional words, etc., now, even though the two versions of the text-based document are actually identical, there is nothing in the text-based document that can be used to hash and deduplicate the data in the text-based document. Each of the multiple hash engines 315 can be configured differently so that each hash engine separately hashes the original text-based document and the new version of the text-based document for deduplication.

[0046] Each of the hash engines 315a, 315b, 315n can be configured to any hash block size. The hash block sizes of the hash engines 315a, 315b, 315n can be different, such as Figure 4 For example, the first hash engine 315a may hash data with a first hash block size, the second hash engine 315b may hash data with a second hash block size, and the "nth" hash engine 315n may hash data with a third hash block size. In addition, each hash engine 315a, 315b, 315n may be configured to start hashing the data set at an offset, as shown. Figure 4 For example, a first hash engine 315a may hash data using a first offset, a second hash engine 315b may hash data using a second offset, and an "nth" hash engine 315n may hash data using a third or "nth" offset.

[0047] In another example, Figure 4As shown, to maintain 32k-bit blocks and a 4k-bit hash block size for a data set, the first hash engine 315a may hash the data set using a first hash block configuration 410 (e.g., a 4k-bit hash block size and an offset of zero), the second hash engine 315b may hash the data set using a second hash block configuration 420 (e.g., a 4k-bit hash block size and a 1k-bit offset), the third hash engine (not shown) may hash the data set using a third hash block configuration 430 (e.g., a 4k-bit hash block size and a 2k-bit offset), and the nth hash engine 315n may hash the data set using a fourth hash block configuration 440 (e.g., a 4k-bit hash block size and a 3k-bit offset). As described in the above example, the hash block sizes are the same across the hash block configurations, but the offsets are different. Alternatively, the hash block sizes within the hash block configurations may differ, and the offsets may differ. Alternatively, the hash block sizes within the hash block configurations may differ, but the offsets may be the same. Another parameter of the hash block configuration may include a hash algorithm. When hashing the same data set, each hash engine 315 may use the same or different hash algorithms.

[0048] In embodiments, the dataset size, hash block size, and hash engine offset do not need to be the same or on byte boundaries. In at least one embodiment, the hash engine can stop after "x" hashes and can return a hash across the entire "hash block." Additionally or alternatively, the hash engine can continue and return a hash across the remainder of the "partial hash block" at the end of the command. For stateful operations on slice hashes, the last hash option can be beneficial because hashing on a partial hash block at the end of a command can be resumed, continued, and completed in a "next" command that is statefully appended to the current command.

[0049] In another example, each hash engine 315 can start at bit zero, and each hash engine 315 can use an independent algorithm. Hashes can be stored under different algorithms, which can be useful for verification or checking for false duplicate hits. Another benefit of using multiple hash engines 315 to hash the same data set in a single pass can help dedupe data in multiple types of systems, such as older systems that use one type of hash block configuration and newer systems that use another type of hash block configuration (e.g., a more advanced hash). This flexibility in hash block configuration can allow multiple hash engines 315 to generate multiple hashes, so that even in a cluster where various storage devices with various hash tables may exist, multiple hash engines 315 can generate hashes for data in a single pass without having to run multiple times. Additionally or alternatively, existing systems can migrate from a smaller hash algorithm to a newer or larger hash algorithm to improve deduplication or increase capacity for conflicts that may increase with older algorithms. In at least one embodiment, the hash block configuration can be set globally or independently for each hash engine 315. The hash engine 315 may store the various hashes in a hash index 140 , which may be accessed by the data source 105 .

[0050] In at least one embodiment, the hashes of each hash engine 315 can be analyzed to identify hash hits. The number of hash hits can vary between hash engines 315 because hash engines 315 use different hash block configurations. In at least one embodiment, the hash corresponding to the greatest number of hash hits can be selected for deduplication, thereby improving the performance of the deduplication system.

[0051] In at least one embodiment, duplicate data may cross hash boundaries, meaning that a first portion of the duplicate data is part of a first hash of a data set and a second portion of the duplicate data is part of a second hash of the data set. In this case, the hash corresponding to the greatest number of hash hits or hash bytes may be selected for deduplication.

[0052] In cases where no duplicate data overlaps a hash boundary, the data set can be deduplicated using hashes from all of the multiple hash engines 315. When hashes from fewer than all of the hash engines 315 overlap, one of the overlapping sets can be selected, while the others can be discarded. For example, if each of five hash engines 315 hashes a data set in a single pass, and there is overlap between three of the five hash engines 315, one of the three overlapping hash engines and the two hash engines without overlap can be selected. The selected hash can be used to deduplicate the data set. Deduplicating data in this manner can improve both deduplication and compression rates.

[0053] In at least one embodiment, as combined Figure 1 As discussed in more detail, the deduplication system 300 may also use a hash block size that is different from the compressed block size. Figure 1 As further described, the data deduplication system 300 can employ skipping techniques when hashing. For example, each of the plurality of hash engines 315 can identify data to be skipped when hashing a data set. Each of the plurality of hash engines 315 can skip the same data, or alternatively, can skip different data. The plurality of hash engines 315 can skip different data when, for example, the skippable data falls on a hash boundary. In at least one embodiment, the plurality of hash engines 315 identify the data to be skipped before creating the hash boundary.

[0054] Modifications, additions, or omissions may be made to the deduplication system 300 without departing from the scope of the present disclosure.

[0055] Figure 4 A data set 400 is shown that can be hashed by different hash engines, each having a different hash block configuration 410, 410, 420, 430, 440, and compressed with a compressed block size 450. The hash block configuration may include various adjustable parameters, such as hash block size, offset, algorithm, etc. In at least one embodiment, the hash block size can be completely independent of the other hash engines, the data set size, and / or the compressed block size. In at least one embodiment, the hash block size of different hash blocks can be variable. Embodiments that would utilize this capability may include a hash algorithm employed by one or more engines that uses rolling hashing to find a margin at which to start hashing. An exemplary hash algorithm may include a Rabin-Karp hashing algorithm. For example, each engine may have a different margin value or be programmed to not start at the same margin to find an optimal margin.

[0056] Figures 5 to 7 Flowcharts showing exemplary methods related to hashing and deduplication are provided. These methods may be performed by processing logic comprising hardware (circuitry, dedicated logic, etc.), software (e.g., software running on a general purpose computer system or a dedicated machine), or a combination of both. The processing logic may include: Figure 1 and Figure 3The deduplication manager 110, hash engine 115 (or hash engine 315), compression manager 125 and / or accelerator card 155 or another computer system or device. However, another system or a combination of multiple systems can be used to perform these methods. For simplicity of explanation, the methods described herein are depicted and described as a series of actions. However, the actions according to the present disclosure can occur in various orders and / or simultaneously, and can occur together with other actions not presented and described herein. In addition, not all of the actions shown can be used to implement the methods according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that these methods can alternatively be represented as a series of interrelated states through state diagrams or events. In addition, the methods disclosed in this specification can be stored on an article of manufacture, such as a non-transitory computer-readable medium, to facilitate the transmission and transfer of these methods to a computing device. The term article of manufacture used herein is intended to include computer programs that can be accessed from any computer-readable device or storage medium. Although shown as discrete blocks, the various blocks can be divided into additional blocks, combined into fewer blocks, or eliminated depending on the desired implementation.

[0057] Figure 5 A flow chart of an exemplary method 500 for hashing a data set is shown. Method 500 may begin at block 505, where processing logic may receive user input indicating a hash block size via a user interface. At block 510, processing logic may set the hash block size based on the user input.

[0058] At block 515, processing logic may identify a data set to be deduplicated (based on the hash block size) and compressed (based on the compressed block size). The hash block size may be smaller than the compressed block size. The compressed block size may be at least twice the hash block size. The hash block size may be a divisor of the compressed block size. The data set may include an electronic file. The compressed block size may be equal to the size of the electronic file. At block 520, processing logic may define a set of data blocks within the data set based on the hash block size.

[0059] At block 525, processing logic may generate a hash for each data block in the set of data blocks within the data set. Additionally or alternatively, processing logic may generate a first instruction to generate a hash for each data block in the set of data blocks within the data set.

[0060] At block 530, processing logic may dedupe the data blocks within the data set based on the corresponding hashes of the data blocks. In at least one embodiment, the deduplicated data blocks are not compressed. Additionally or alternatively, processing logic may generate a second instruction to dedupe the data blocks within the data set based on the corresponding hashes of the data blocks.

[0061] At block 535, processing logic may compress the data set based on the compression block size. In at least one embodiment, the compression block size may not be an integer multiple of the hash block size. A second data block in the set of data blocks may have a unique hash block size. Compressing the data set based on the compression block size may include at least one of refraining from compressing the second data block or discarding a hash associated with the second data block. Additionally or alternatively, processing logic may generate a third instruction to compress the data set based on the compression block size.

[0062] Figure 6 A flow chart of another exemplary method 600 for hashing a data set is shown. The method 600 may begin at block 605, where processing logic may identify a data set to be hashed, the data set comprising a set of data chunks.

[0063] At block 610, processing logic may identify data to be skipped within the data set. In at least one embodiment, the data to be skipped within the data set includes at least one of the following: data characterized as a pattern, a header, a footer, non-volatile memory express (NVMe) protection data, or error recovery control data. In at least one embodiment, identifying the data to be skipped within the data set includes: identifying skip candidates within the data set; providing the skip candidates to be provided via an interface; and receiving confirmation of the skip candidates.

[0064] At block 615, processing logic may generate a hash for each data block in the set of data blocks within the data set, excluding data to be skipped. In at least one embodiment, identifying the data to be skipped within the data set includes determining that the data includes a repeating field. The repeating field may include a specific byte count. In at least one embodiment, generating a hash for each data block in the set of data blocks within the data set, excluding data to be skipped, includes identifying a first byte of the field and skipping a number of bytes equal to the byte count. In at least one embodiment, generating a hash for each data block in the set of data blocks within the data set, excluding data to be skipped, includes skipping the data based on an offset and a count.

[0065] At block 620, processing logic may compress the data using a compressed block size. At block 625, processing logic may provide the skipped data to a record for storage. At block 630, processing logic may provide a record of the skipped data via a user interface. In at least one embodiment, the source data may not be skipped, and all hashing, including offsets, may be performed on the compressed data.

[0066] Figure 7 A flow chart of another exemplary method 700 for hashing a data set is shown. The method 700 may begin at block 705, where processing logic may identify a data set to be hashed. The data set may include a set of data chunks.

[0067] At block 710, processing logic may generate a first hash for each data block in a set of data blocks within a data set using a first hash engine. In at least one embodiment, the first hash engine may be configured to generate the first hash using a first hash algorithm. In at least one embodiment, the first hash engine may be configured to generate the first hash based on a first hash block size. In at least one embodiment, the first hash engine may be configured to begin generating the first hash at the first bit of the data set.

[0068] At block 715, processing logic may use a second hash engine to generate a second hash for each data block in the set of data blocks within the data set. In at least one embodiment, the second hash engine may be configured to generate the second hash using a second hash algorithm. In at least one embodiment, the second hash engine may be configured to generate the second hash based on a second hash block size. In at least one embodiment, the second hash engine may be configured to generate the second hash at a second bit in the data set. The first bit in the data set may be bit zero, and the second bit in the data set may be determined based on an offset from the first bit. In at least one embodiment, processing logic may receive a selection of the first hash engine and the second hash engine from a set of available hash engines.

[0069] At block 720, processing logic may deduplicate a particular data block within the data set based on a particular first hash associated with the particular data block or a particular second hash or subsequent hash associated with the particular data block. At block 725, processing logic may compress the data set based on the compression block size. In at least one embodiment, hashing may be performed after data compression.

[0070] Figure 8 A diagrammatic representation of a machine is shown in the example form of a computing device 800, within which a set of instructions can be executed for causing the machine to perform any one or more of the methods discussed herein. The computing device 800 may include a rack server, a router computer, a server computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, or any computing device with at least one processor, in which a set of instructions can be executed for causing the machine to perform any one or more of the methods discussed herein. In alternative embodiments, the machine can be connected (e.g., networked) to other machines in a local area network, an intranet, an extranet, or the Internet. The machine can operate in the capacity of a server machine in a client-server network environment. Furthermore, although a single machine is shown, the term "machine" may also include any collection of machines that, individually or in combination, execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0071] The example computing device 800 includes a processing device (e.g., a processor) 802, a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory 806 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 816 that communicate with each other via a bus 808.

[0072] The processing device 802 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. More specifically, the processing device 802 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing device 802 may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 802 is configured to execute instructions 826 to perform the operations and steps discussed herein.

[0073] The computing device 800 may also include a network interface device 822 that can communicate with the network 818. The computing device 800 may also include a display device 810 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 812 (e.g., a keyboard), a cursor control device 814 (e.g., a mouse), and a signal generating device 820 (e.g., a speaker). In at least one embodiment, the display device 810, the alphanumeric input device 812, and the cursor control device 814 may be combined into a single component or device (e.g., an LCD touch screen).

[0074] The data storage device 816 may include a computer-readable storage medium 824 on which is stored one or more sets of instructions 826 embodying any one or more of the methodologies or functionality described herein. The instructions 826 may also reside, completely or at least partially, in the main memory 804 and / or the processing device 802 during execution of the instructions by the computing device 800, with the main memory 804 and the processing device 802 also constituting computer-readable media. Instructions may also be transmitted or received over the network 818 via the network interface device 822.

[0075] Although the computer-readable storage medium 826 is shown as a single medium in the exemplary embodiment, the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" may also include any medium that can store, encode, or carry a set of instructions executed by a machine and cause the machine to perform any one or more of the methods of the present disclosure. Therefore, the term "computer-readable storage medium" may be understood to include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0076] As is customary, the various features shown in the drawings may not be drawn to scale. The illustrations presented in this disclosure are not intended to be actual views of any particular device (e.g., equipment, system, etc.) or method, but are merely idealized representations for describing various embodiments of the present disclosure. Therefore, for the sake of clarity, the sizes of various features may be arbitrarily expanded or reduced. In addition, some drawings may be simplified for the sake of clarity. Therefore, the drawings may not depict all components of a given device (e.g., equipment) or all operations of a particular method.

[0077] The terms used herein, especially in the claims (e.g., the bodies of the claims), are generally considered to be “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” etc.).

[0078] Furthermore, if a specific number of claim recitations is intended, such intent will be expressly recited in the claim, and in the absence of such recitation, such intent is absent. For example, to aid understanding, a claim may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be interpreted as implying that a claim recitation introduced by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" as well as, for example, "a" or "an" (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"); the same is true for the use of definite articles used to introduce claim recitations.

[0079] Furthermore, even if a specific number of an introduced claim recitation is explicitly recited, it should be understood that such recitation should be interpreted to mean at least the recited number (e.g., the simple recitation of "two recitations" without other modifiers means at least two recitations, or two or more recitations). Moreover, in the case of conventions like "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc.", generally speaking, such constructions are intended to include A alone, B alone, C alone, both A and B, both A and C, both B and C, or all three of A, B, and C, etc. For example, the term "and / or" should be interpreted in this manner.

[0080] Furthermore, any discrete word or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of one, either, or both terms. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B."

[0081] In addition, the terms "first", "second", "third", etc. are not necessarily used herein to imply a particular order or number of elements. Typically, the terms "first", "second", "third", etc. are used as general identifiers to distinguish different elements. If there is no indication that the terms "first", "second", "third", etc. imply a particular order, these terms should not be understood to imply a particular order. In addition, if the terms "first", "second", "third", etc. are indicated to imply a particular number of elements, these terms should not be understood to imply a particular number of elements. For example, a first widget may be described as having a first side, and a second widget may be described as having a second side. Use of the term "second side" with respect to the second widget may be to distinguish this side of the second widget from the "first side" of the first widget, rather than to imply that the second widget has two sides.

[0082] All examples and conditional language described herein are for illustrative purposes to help readers understand the present invention and the concepts contributed by the inventors to advance the art, and should be interpreted as not being limited to these specific examples and conditions. Although the embodiments of the present disclosure have been described in detail, it should be understood that various changes, substitutions, and modifications may be made thereto without departing from the spirit and scope of the present disclosure.

Claims

1. A hashing method for deduplicating data by skipping selected data, comprising: identifying a data set to be hashed, the data set comprising a set of data chunks; identifying data to be skipped within the data set; generating, by a hash engine, a hash for each data block in the set of data blocks in the data set excluding the data to be skipped, wherein generating the hash for each data block in the set of data blocks in the data set excluding the data to be skipped comprises skipping a number of bytes equal to a particular byte count; and The data is compressed.

2. The method according to claim 1, wherein Identifying data to be skipped within the data set includes determining that the data includes repeated fields.

3. The method according to claim 2, wherein: The repeating field has a specific byte count, wherein generating a hash for each data block in the set of data blocks within the data set excluding the data to be skipped includes identifying a first byte of the repeating field and skipping a number of bytes equal to the specific byte count.

4. The method according to claim 1, further comprising: providing the skipped data to the record for storage; as well as The record of the skipped data is provided through a user interface.

5. The method according to claim 1, wherein The data to be skipped within the data set includes at least one of the following data: data characterized as a pattern, non-volatile memory express NVMe protection data, or error recovery control data.

6. The method according to claim 1, wherein Identifying data to be skipped within the data set includes: identifying skip candidates within the dataset; providing the skip candidates to be provided through an interface; and Confirmation of the skip candidate is received.

7. A hashing system for deduplicating data by skipping selected data, comprising: Memory; as well as a processing unit coupled to the memory and configured to cause the system to perform operations comprising: identifying a data set to be hashed, the data set comprising a set of data chunks; identifying data to be skipped within the data set; generating, by a hash engine, a hash for each data block in the set of data blocks in the data set excluding the data to be skipped, wherein generating the hash for each data block in the set of data blocks in the data set excluding the data to be skipped comprises skipping a number of bytes equal to a particular byte count; and The data is compressed.

8. The system according to claim 7, wherein: Identifying data to be skipped within the data set includes determining that the data includes a repeated field, wherein the repeated field has a particular byte count.

9. The system according to claim 7, wherein: The operations further include: providing the skipped data to the record for storage; and The record of the skipped data is provided through a user interface.

10. The system according to claim 7, wherein: The data to be skipped within the data set includes at least one of the following data: data characterized as a pattern, a header, a footer, non-volatile memory express NVMe protection data, or error recovery control data.

11. The system according to claim 7, wherein: Identifying data to be skipped within the data set includes: identifying skip candidates within the dataset; providing the skip candidates to be provided through an interface; and Confirmation of the skip candidate is received.

12. A hashing system for deduplicating data by skipping selected data, comprising: Memory; as well as One or more processors, coupled to the memory, configured to perform operations including: identifying a data set to be hashed, the data set comprising a set of data chunks; identifying data to be skipped within the data set; generating a hash for each data block in the set of data blocks in the data set excluding the data to be skipped, wherein generating a hash for each data block in the set of data blocks in the data set excluding the data to be skipped comprises skipping a number of bytes equal to a particular byte count; and The data is compressed.

13. The system according to claim 12, wherein: Identifying data to be skipped within the data set includes determining that the data includes repeated fields.

14. The system according to claim 13, wherein: The repeating field has a specific byte count, wherein generating a hash for each data block in the set of data blocks excluding the data to be skipped in the data set includes identifying a first byte of the repeating field and skipping a number of bytes equal to the specific byte count.

15. The system according to claim 12, wherein: Generating a hash for each data block in the set of data blocks except for the data to be skipped within the data set includes skipping data based on an offset and a skip count.

16. The system of claim 12, wherein: The data to be skipped within the data set includes at least one of the following data: data characterized as a pattern, a header, a footer, non-volatile memory express NVMe protection data, or error recovery control data.

17. The system of claim 12, wherein: Identifying data to be skipped within the data set includes: identifying skip candidates within the dataset; providing the skip candidates to be provided through an interface; and Confirmation of the skip candidate is received.

18. The system according to claim 15, wherein: The offset is a tunable parameter.

Images