Method, storage system, and machine-readable medium for data reduction

Abstract

The present disclosure relates to data reduction in a storage system. Example embodiments relate to storing data in a storage system. Examples include receiving, by a storage controller of a storage system, a data unit to be stored in a persistent storage device of the storage system. The storage controller determines a maximum entropy value and a minimum entropy value for the received data unit. The storage controller determines whether the received data unit is data reducible based on at least the minimum entropy value and the maximum entropy value. In response to determining that the received data unit is data reducible, the storage controller performs at least one reduction operation on the received data unit.

Description

Technical Field

[0001] This disclosure generally relates to data reduction techniques. Background Art

[0002] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Some example data reduction techniques include data compression and deduplication. Deduplication identifies duplicate data units and attempts to reduce or eliminate the number of instances of duplicate data units stored in a storage system. Summary of the Invention

[0003] According to some embodiments of the present disclosure, a storage system is provided, comprising a processor and a machine-readable storage device storing instructions, the instructions being executable by the processor to perform the following operations: receiving a data unit to be stored in a persistent storage device of the storage system; determining a first entropy value for the received data unit; determining a second entropy value for the received data unit; comparing the first entropy value with a first lower threshold; comparing the second entropy value with a first upper threshold; and in response to at least one of determining that the first entropy value is not less than the first lower threshold and determining that the second entropy value is not less than the first upper threshold: comparing the first entropy value with a second lower threshold; comparing the second entropy value with a second upper threshold; and in response to at least one of determining that the first entropy value is not less than the second lower threshold and determining that the second entropy value is not less than the second upper threshold: storing the data unit in the persistent storage device without attempting to reduce the data unit.

[0004] According to some embodiments of the present disclosure, a non-transitory machine-readable medium storing instructions is provided, which, when executed, causes a processor to perform the following operations: receive a data unit to be stored in a persistent storage device of a storage system; determine a Rayleigh entropy value of the received data unit using a first technique; determine a Hartley entropy value of the received data unit using a second technique different from the first technique; determine whether the received data unit is capable of data reduction based at least on the Rayleigh entropy value of the received data unit and the Hartley entropy value of the received data unit; and in response to determining that the received data unit is capable of data reduction, perform at least one reduction operation on the received data unit.

[0005] According to some embodiments of the present disclosure, a method for data reduction is provided, comprising: receiving, by a storage controller of a storage system, a data unit to be stored in a persistent storage device of the storage system; determining, by the storage controller, a first entropy value of the received data unit using a first technique; determining, by the storage controller, a second entropy value of the received data unit using a second technique different from the first technique; determining, by the storage controller, whether the received data unit is capable of a data compression operation based at least on the first entropy value of the received data unit and the second entropy value of the received data unit; and in response to determining, based on the first entropy value and the second entropy value, that the received data unit is capable of data reduction, performing, by the storage controller, the data compression operation on the received data unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Some embodiments are described with respect to the following figures.

[0007] Figure 1A to Figure 1B is a schematic diagram of an example storage system according to some embodiments.

[0008] Figures 2A to 2B is an illustration of an example data structure according to some implementations.

[0009] Figure 3 is an illustration of an example process according to some implementations.

[0010] Figures 4A to 4B is an illustration of an example process according to some implementations.

[0011] Figure 5 is an illustration of an example process according to some implementations.

[0012] Figure 6 is a diagram of an example machine-readable medium having instructions stored thereon, according to some implementations.

[0013] Figure 7 is a schematic diagram of an example computing device in accordance with some embodiments.

[0014] Figure 8 is an illustration of an example process according to some implementations.

[0015] Throughout the drawings, like reference numerals refer to similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be exaggerated to more clearly illustrate the examples shown. Furthermore, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0016] In the present disclosure, the use of the terms "a," "an," or "the" is intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising" or "have / having" specify the presence of stated elements, but do not preclude the presence or addition of other elements.

[0017] In some examples, the storage system may perform a data reduction operation to attempt to reduce the amount of space required to store the received data. The data reduction operation may include a data compression operation, a data deduplication operation, or a combination thereof. For example, the compression operation may include encoding information using fewer bits than used in its original form. Further, the data deduplication operation may include dividing the incoming data stream into a plurality of data units and determining which incoming data units are duplicates of previously stored data units. If the incoming data unit is determined to be a duplicate, the storage system may store a reference to the previous data unit instead of storing the duplicate incoming data unit.

[0018] In some examples, the deduplication operation can include comparing the fingerprint of the incoming data unit with the fingerprint of the stored data unit. As used herein, the term "fingerprint" refers to a value obtained by applying a function to the contents of a data unit (where "content" can include all or a subset of the contents of a data unit). Examples of functions that can be applied include hash functions that generate a hash value based on the incoming data unit. Examples of hash functions include cryptographic hash functions such as the Secure Hash Algorithm 2 (SHA-2) hash function (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions can be used.

[0019] A "storage system" may include a storage device or an array of storage devices. A storage system may also include (multiple) storage controllers that manage access to (multiple) storage devices. A "data unit" may refer to any portion of data that can be individually identified in a storage system. In some cases, a data unit may refer to a chunk, a collection of chunks, or any other portion of data. In some examples, a storage system may store data units in a persistent storage device. The persistent storage device may be implemented using one or more (multiple) persistent (e.g., non-volatile) storage devices, such as (multiple) disk-based storage devices (e.g., (multiple) hard disk drives (HDDs)), (multiple) solid-state devices (SSDs) (e.g., (multiple) flash memory devices), etc., or a combination thereof.

[0020] A "controller" may refer to a hardware processing circuit, which may include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuits. Alternatively, a "controller" may refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) executable on the hardware processing circuit.

[0021] In some examples, a deduplication storage system may receive and store various types of data (e.g., from different applications, having different types of content, etc.). However, some types of data may not be reducible. For example, reducible data may include data that does not contain repeating patterns, data that is substantially random, etc. Furthermore, attempting to reduce such data may consume a relatively large amount of processing time and / or energy, but may provide little or no benefit in reducing the storage space required to store the received data. Therefore, attempting to reduce reducible data may result in relatively slow and / or inefficient performance of the storage system.

[0022] According to some embodiments of the present disclosure, a storage system may include a storage controller for determining whether to attempt data reduction on a particular data unit. In some embodiments, the storage controller may analyze an incoming data unit to determine one or more entropy values ​​for the data unit, and may determine that the data unit is reducible when the entropy value satisfies a corresponding threshold. If it is determined that the data unit is reducible, the storage controller may attempt to reduce the data unit. In this way, the storage controller may avoid attempting to reduce data units that are not reducible. Therefore, the storage controller may save processing time and / or energy that would otherwise be wasted in unproductive reduction attempts, and may thereby improve the performance and efficiency of the storage system.

[0023] Figure 1A to Figure 1B - Sample Storage System

[0024] Figure 1A An example of a storage system 100 including a storage controller 110, a memory 115, and a persistent storage device 140 according to some embodiments is shown. As shown, the persistent storage device 140 may include reduced data 150 and unreduced data 160. The persistent storage device 140 may include one or more non-transitory storage media, such as a hard disk drive (HDD), a solid-state drive (SSD), an optical disk, or a combination thereof. The memory 115 may be implemented using a semiconductor memory such as a random access memory (RAM).

[0025] In some embodiments, the storage controller 110 may include or execute data reduction logic 120 and filter logic 130. The storage controller 110 and the included logic 120, 130 may be implemented as a processor that executes instructions (e.g., software and / or firmware) stored in a machine-readable storage medium, in hardware (e.g., circuitry), or the like.

[0026] In some implementations, the data reduction logic 120 may perform a reduce operation to attempt to reduce the selected data unit 105. For example, Figure 1B As shown, data reduction logic 120 may include deduplication logic 122 and compression logic 126. Deduplication logic 122 may perform a deduplication operation, which includes generating a fingerprint (e.g., a complete or partial hash value) for a received data unit 105 and comparing the generated fingerprint to a stored fingerprint of a previously received data unit. If the comparison results in a match, it may be determined that the storage system 100 already stores a replica of the received data unit 105. Further, in some embodiments, compression logic 126 may perform data compression of the data unit. For example, compression logic 126 may implement one or more of run-length encoding, Huffman encoding, arithmetic encoding, a Lempel-Ziv algorithm, and the like.

[0027] In some embodiments, the filter logic 130 may determine whether a received data unit 105 is reducible and, if so, may cause the data reduction logic 120 to attempt to reduce the data unit 105. For example, the filter logic 130 may determine one or more numerical values ​​indicative of an entropy characteristic of the data unit 105 (referred to herein as "entropy values") and may compare the entropy values ​​to corresponding thresholds 137 to determine whether the data unit 105 is reducible. If the comparison of the entropy values ​​to the corresponding thresholds 137 indicates that the data unit 105 is reducible, the filter logic 130 may cause the data reduction logic 120 to attempt to reduce the data unit 105. If the reduction is successful, the reduced data unit 105 may be stored in the persistent storage 140 as reduced data 150. However, if the comparison of the entropy values ​​to the thresholds 137 indicates that the data unit 105 is not reducible, no reduction attempt is made on the data unit 105, and the data unit 105 is stored in the persistent storage 140 as unreduced data 160. In this manner, the filter logic 130 may reduce processing time and / or energy wasted in attempting to reduce irreducible data units 105 and may thereby improve performance of the storage system 100 .

[0028] In some examples, a reduce operation performed by data reduction logic 120 may fail to reduce the size of data unit 105 by at least a minimum amount or ratio (e.g., at least a 2:1 space reduction). Such a reduce operation may be referred to herein as a "failed" reduce. The minimum amount of reduction required by storage system 100 may be specified as a system configuration setting. In some embodiments, upon detecting a failed reduction of a data unit 105, data reduction logic 120 may store the data unit 105 in an unreduced form in persistent storage 140 (e.g., as unreduced data 160).

[0029] In some embodiments, the filter logic 130 may use multiple entropy values, including a maximum entropy value, a minimum entropy value, and a Shannon entropy value. As used herein, the term "maximum entropy" may refer to a value that measures the maximum possible degree of uncertainty present in a set of data samples. Further, as used herein, the term "minimum entropy" may refer to a value that measures the minimum possible degree of uncertainty present in a set of data samples.

[0030] In some examples, a discrete random variable X may have n possible values, where the probability of the i-th outcome is p i In such examples, the maximum entropy (also known as Hartley entropy) can be calculated as the binary logarithm of the number of values ​​that X can occur with non-zero probability, which can be expressed as log2n. Further, in some examples, the minimum entropy (also known as Rayleigh entropy) can be calculated as the negative binary logarithm of the probability of the most likely outcome, which can be expressed as -log2(max p i ). In addition, in some examples, the Shannon entropy can be calculated as the probability of each outcome multiplied by the negative of the sum of the binary logarithms of the probability of each outcome, which can be expressed as -∑n(p i *log2(p i )).

[0031] In some embodiments, the filter logic 130 may use modified calculations to calculate the maximum entropy value, the minimum entropy value, and the Shannon entropy value. For example, the modified calculations performed by the filter logic 130 may use integer arithmetic to calculate the maximum entropy value, the minimum entropy value, and the Shannon entropy value. Figures 4A to 4B 1 to discuss an example of a modified calculation performed by the filter logic 130. In another example, the modified calculation uses a lookup table 135 to calculate the maximum entropy value, the minimum entropy value, and the Shannon entropy value. The lookup table 135 can be a stored data structure that includes an array of constants for determining binary logarithmic values. Figure 2ADiscussing example implementations of the lookup table 135. In some implementations, the modified calculations performed by the filter logic 130 can avoid performing floating point arithmetic and direct calculation of binary logarithms. Thus, using the modified calculations can reduce the processing load associated with using the filter logic 130.

[0032] In some embodiments, the filter logic 130 may periodically identify data units 105 having entropy values ​​within a given amount of a corresponding threshold 137 (e.g., not satisfying the threshold 137, but within a specific distance of the threshold 137). The filter logic 130 may attempt to reduce the identified data unit 105 and may determine whether the attempted reduction failed. Upon determining that the attempted reduction did not fail, the filter logic 130 may adjust the threshold 137 so that the entropy value of the data unit 105 satisfies the adjusted threshold 137. In this manner, the filter logic 130 may periodically adjust the threshold 137 to better correspond to the entropy level associated with the data units that may be reduced, and may thereby improve data reduction performance in the storage system 100.

[0033] Figures 2A to 2B - Example data structure

[0034] Figure 2A An example lookup table 135 is shown according to some embodiments. As shown, the lookup table 135 may include multiple entries, where each entry corresponds to a different combination of an index value I and a constant value C. In some embodiments, the filter logic 130 ( Figure 1A As shown in FIG. 1 , the input can be matched with the index value I of the lookup table 135, and the constant value C corresponding to the input value can be determined accordingly. Figure 2A The index value I is shown as a data field stored in the lookup table 135, but other implementations are possible. For example, the lookup table 135 can be implemented as an ordered array of constants, and the index value I for a given constant C can be determined by the specific position of the constant C in the ordered array (e.g., the first position has index I=0, the second position has index I=1, etc.).

[0035] In some embodiments, the index value I of the lookup table 135 can correspond to a defined range of integer values ​​(e.g., 0 to 256). Further, for each index value I, the corresponding constant C can be a floating point number of a given length (e.g., seven digits) that is calculated as the binary logarithm of the index value I (i.e., log2(I)). Thus, in such embodiments, the binary logarithm of I can be obtained by performing a lookup of the input I in the lookup table 135. In this manner, using the lookup table 135 can allow the logarithm value to be determined without using floating point arithmetic, and thus can reduce the amount of processing required by the filter logic 130 to determine the entropy value.

[0036] Figure 2B An example embodiment of a data structure storing threshold values ​​137 is shown. In some embodiments, the threshold values ​​137 may include, in order of increasing value, a first lower threshold value (LT1), a second lower threshold value (LT2), a third threshold value (T3), a first upper threshold value (UT1), and a second upper threshold value (UT2). However, other embodiments are possible. The threshold values ​​137 may be compared to corresponding entropy values ​​generated by the filter logic 130 to determine whether a given data unit is eligible for data reduction. hereinafter, with reference to Figure 3 An example process of comparing the threshold value 137 to the corresponding entropy value is discussed.

[0037] Figure 3 - Example process for storing data

[0038] Figure 3 An example process 300 for storing data according to some embodiments is shown. In some examples, the storage controller 110 ( Figure 1A 300. The process 300 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustration purposes, reference may be made to an example of an embodiment according to some embodiments. Figures 1A to 2B Details are described for process 300. However, other implementations are possible.

[0039] Block 310 may include receiving a data unit to be stored in a persistent storage device. Block 320 may include determining a minimum entropy value (MIN) for the received data unit. Block 330 may include determining a maximum entropy value (MAX) for the received data unit. After block 330, process 300 may continue at decision block 340 (described below). For example, referring to Figure 1A , the storage controller 110 receives the inbound data stream and divides the data stream into separate data units. The filter logic 130 performs modified calculations to determine the maximum entropy value and the minimum entropy value. In some embodiments, these modified calculations are performed using a lookup table 135. Figure 4A Discusses example procedures for calculating maximum and minimum entropy values.

[0040] Decision block 340 may include determining whether the minimum entropy value (MIN) is less than a first lower threshold value (LT1), and whether the maximum entropy value (MAX) is less than a first upper threshold value (UT1). After a positive determination ("yes") is obtained at decision block 340, process 300 may continue at block 380, including attempting data reduction on the data unit. For example, referring to Figure 1A , the storage controller 110 determines that the minimum entropy value (MIN) is less than a first lower threshold value (eg, LT1 is equal to 4.0, as Figure 2B As shown), and further determining that the maximum entropy value (MAX) is less than the first upper threshold value (eg, UT1 is equal to 6.0, as shown Figure 2B In response to these determinations, storage controller 110 (e.g., using deduplication logic 122 and / or compression logic 126) performs one or more data reduction operations for data unit 105. If the reduction of data unit 105 is successful, data unit 105 is stored in persistent storage 140 as reduced data 150.

[0041] However, when a negative determination ("No") is obtained at decision block 340 (i.e., if the minimum entropy value (MIN) is not less than the first lower threshold value (LT1), or if the maximum entropy value (MAX) is not less than the first upper threshold value (UT1)), process 300 can continue at decision block 350, including determining whether the minimum entropy value (MIN) is less than the second lower threshold value (LT2), and whether the maximum entropy value (MAX) is less than the second upper threshold value (UT2). After a positive determination ("Yes") is obtained at decision block 350, process 300 can continue at block 360, including determining the Shannon entropy value (SE) of the received data unit. After block 360, process 300 can continue at decision block 370 (as described below). However, when a negative determination ("No") is obtained at decision block 350 (i.e., if the minimum entropy value (MIN) is not less than the second lower threshold value (LT2), or if the maximum entropy value (MAX) is not less than the second upper threshold value (UT2)), process 300 can continue at block 390 (as described below). For example, reference Figure 1A , the storage controller 110 determines that the minimum entropy value (MIN) is less than a second lower threshold value (eg, LT2 is equal to 5.0, such as Figure 2B As shown), and further determining that the maximum entropy value (MAX) is less than a second upper threshold value (eg, UT2 is equal to 7.5, as shown Figure 2B In response to these determinations, the filter logic 130 performs a modified calculation to determine the Shannon entropy value. In some embodiments, this modified calculation is performed using a lookup table 135. Figure 4B Discusses an example procedure for calculating Shannon entropy.

[0042] Decision block 370 may include determining whether the Shannon entropy value (SE) is less than a third threshold value (T3). Upon a positive determination ("yes") at decision block 370, process 300 may continue at block 380 (as described above). However, upon a negative determination ("no") at decision block 370 (i.e., if the Shannon entropy value (SE) is not less than the third threshold value (T3)), process 300 may continue at block 390, including storing the data unit without performing any data reduction. After either block 380 or block 390, process 300 may be complete. In some examples, process 300 may be repeated for multiple data units 105 received by storage system 100. For example, with reference to Figure 1A , if the storage controller 110 determines that the Shannon entropy value (SE) is less than a third threshold value (eg, T3 is equal to 5.5, as Figure 2B ), the storage controller 110 (e.g., using the deduplication logic 122 and / or the compression logic 126) performs one or more data reduction operations on the data unit 105. If the reduction of the data unit 105 is successful, the data unit 105 is stored in the persistent storage 140 as reduced data 150. Otherwise, if the Shannon entropy value (SE) is not less than a third threshold, the storage controller 110 does not reduce the data unit 105 and stores the data unit 105 in the persistent storage 140 as unreduced data 160. Further, if the reduction of the data unit 105 is unsuccessful, the storage controller 110 stores the data unit 105 in the persistent storage 140 as unreduced data 160.

[0043] Figures 4A to 4B - Example process for calculating entropy value

[0044] Figure 4A An example process 400 for calculating maximum entropy and minimum entropy values ​​is shown. Figure 4B An example process 405 for calculating Shannon entropy values ​​is shown. In some examples, the storage controller 110 ( Figure 1A 400, 405. The processes 400, 405 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustration purposes, reference may be made to an example of an embodiment according to some embodiments. Figures 1A to 2B Details of the processes 400, 405 are described. However, other implementations are possible.

[0045] Now refer to Figure 4A, block 410 may include initializing a maximum probability variable ("Max_Prob") to a value of 1. Block 420 may include initializing a cardinality variable ("Cardinality") to a value of 0. Block 430 may include obtaining N samples from the data unit. For example, referring to Figure 1A , the memory controller 110 may generate 8 samples of randomly selected bytes, where each sample is taken from each of 32 evenly distributed blocks within a 16384-byte data unit (eg, a cache memory page).

[0046] At block 440, a loop (defined by blocks 440 through 448) may be entered to process each sample in the data unit (obtained at block 430). Block 442 may include determining the value I of the current sample. Block 444 may include determining a count of sample values ​​I ("count(I)") (i.e., the number of instances in which value I has occurred so far in the samples of the data unit). If the count of sample values ​​I is currently equal to 0, the cardinality variable may be incremented by 1. In some embodiments, the count of sample values ​​I ("count(I)") may be determined using a histogram function.

[0047] Block 444 may include incrementing the count of sample value 1 by 1. Block 448 may include setting the maximum probability variable equal to the current count of sample value 1 if the count of sample value 1 is greater than the current maximum probability variable. After block 448, the loop may return to block 440 (i.e., process another sample in the data unit). After all samples have been processed at block 440, process 400 may continue at block 450.

[0048] Block 450 may include setting the maximum entropy value ("MAX") equal to the value of the formula Int(A(B-log2(Max_Prob))) (i.e., an integer cast of the product of a constant A multiplied by a constant B minus the difference between the binary logarithm of the maximum probability variable). For example, in some embodiments, variable A may be equal to 10, and variable B may be equal to 8. Further, in some embodiments, the binary logarithm of the maximum probability variable may be determined by performing a lookup of the maximum probability variable in lookup table 135.

[0049] Block 450 may include setting the minimum entropy value ("MIN") equal to the value of the formula Int(A*log2(MaxCardinality)) (i.e., an integer conversion of the product of the constant A times the binary logarithm of the cardinality variable). In some implementations, the binary logarithm of the cardinality variable may be determined by performing a lookup of the cardinality variable in the lookup table 135. After block 460, process 400 may be complete.

[0050] Now refer to Figure 4B At block 470, a loop (defined by blocks 470 to 475) may be entered to process the data units (in Figure 4A Block 475 may include setting the value of a total variable ("Total") equal to the previous value of the total variable plus the value of the formula count(I)*(B-log(count(I))) (i.e., the product of the count of sample value I multiplied by the difference between variable B and the binary logarithm of the count of sample value I). In some embodiments, the binary logarithm of the count of sample value I can be determined by performing a lookup of the count of sample value I in lookup table 135.

[0051] After all sample values ​​I are processed at block 470, process 405 may continue at block 480. Block 480 may include setting the Shannon entropy value (SE) equal to the value of the formula Int((A*Total)>>B) (i.e., an integer conversion of the product of the constant A times the total variable, where the product is bitwise right shifted by B bits). After block 480, process 405 may be complete.

[0052] Figure 5 - Example process for storing data

[0053] Figure 5 An example process 500 for storing data according to some embodiments is shown. In some examples, the storage controller 110 ( Figure 1A 500. The process 500 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustration purposes, reference may be made to an example of an embodiment according to some embodiments. Figures 1A to 2B Details are described for process 500. However, other implementations are possible.

[0054] Block 510 may include calculating an entropy value for the received data unit. Block 510 may include attempting to perform data reduction on a data unit having an entropy value that satisfies an entropy threshold. For example, referring to Figure 1A, storage controller 110 receives an inbound data stream and divides the data stream into individual data units 105. Storage controller 110 performs a modified calculation to determine an entropy value for data unit 105 and compares the entropy value to a corresponding threshold value 137 to determine whether data unit 105 is reducible. If it is determined that data unit 105 is reducible, storage controller 110 attempts to reduce data unit 105. If the reduction of data unit 105 is successful, data unit 105 is stored in persistent storage 140 as reduced data 150. Otherwise, if it is determined that data unit 105 is not reducible, storage controller 110 does not reduce data unit 105 and instead stores it in persistent storage 140 as unreduced data 160. Further, if the reduction of data unit 105 is unsuccessful, storage controller 110 stores data unit 105 in persistent storage 140 as unreduced data 160.

[0055] Decision block 530 may include determining whether a consecutive number of data units that were reduced is greater than a first consecutive threshold value ("CT1"). Upon a negative determination ("No") at decision block 530, process 500 may continue at decision block 550 (as described below). Otherwise, upon an affirmative determination ("Yes") at decision block 530, process 500 may continue at block 535, including suspending calculation of entropy values ​​for subsequent data units. Block 540 may include attempting data reduction on subsequent data units without calculating their entropy values. Block 545 may include, in response to detecting a failed data reduction, resuming calculation of entropy values ​​for subsequent data units. After block 545, process 500 may return to block 510. For example, referring to Figure 1A , the storage controller 110 determines that the consecutive number of data units to be reduced (i.e., the number of consecutive data units that were reduced based on their entropy values) is greater than the CT1 threshold. In response, the storage controller 110 begins attempting to reduce each subsequent data unit 105 without calculating its entropy value. After detecting a failed data reduction (i.e., a data reduction that fails to reduce the size of the data unit 105 by at least a specified ratio) for at least one data unit 105, the storage controller 110 resumes calculating entropy values ​​for subsequent data units 105 and uses the entropy values ​​to determine whether to perform a reduce operation on these data units 105.

[0056] Decision block 550 may include determining whether the consecutive number of unreduced data units is greater than a second consecutive threshold value ("CT2"). Upon a negative determination ("No") at decision block 550, process 500 may return to block 510 (i.e., to continue calculating entropy values ​​for the received data units). Otherwise, upon an affirmative determination ("Yes") at decision block 550, process 500 may continue at block 560, including calculating entropy values ​​for a subset of the data units. Block 570 may include storing the data units without attempting data reduction. Block 580 may include, in response to detecting data unit(s) having entropy values ​​that satisfy the entropy threshold value, resuming calculation of entropy values ​​for all data units. Following block 580, process 500 may return to block 510. For example, referring to Figure 1A , the storage controller 110 determines that the consecutive number of unreduced data units (i.e., the number of consecutive data units that have not been reduced based on their entropy values) is greater than the CT2 threshold. In response, the storage controller 110 begins storing each subsequent data unit 105 without attempting to reduce these data units 105. Further, the storage controller 110 calculates entropy values ​​for a subset of the data units 105 (e.g., one in every fifty data units) and compares the entropy values ​​to the threshold 137 to determine whether any data unit 105 in the subset is reducible. If at least a specified number (e.g., 1, 5, etc.) of the data units 105 are determined to be reducible, the storage controller 110 resumes calculating entropy values ​​for subsequent data units 105 and uses the entropy values ​​to determine whether to perform a reduce operation for these subsequent data units 105.

[0057] In some embodiments, process 500 can provide a feedback mechanism for determining whether to attempt data reduction based on information about the most recent number of data units that were reduced or not reduced. Thus, if data reduction is effective, the use of data reduction can be automatically increased. Furthermore, if data reduction is less effective, the use of data reduction can be automatically decreased. In this way, process 500 can improve the performance of the storage system.

[0058] Figure 6 - Example machine-readable medium

[0059] Figure 6 A machine-readable medium 600 storing instructions 610-650 according to some embodiments is shown. The instructions 610-650 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 600 may be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium.

[0060] Instructions 610 may be executed to receive a data unit to be stored in a persistent storage device of a deduplication storage system. Instructions 620 may be executed to determine a minimum entropy value for the received data unit. Instructions 630 may be executed to determine a maximum entropy value for the received data unit. For example, referring to Figure 1A , the storage controller 110 receives the inbound data stream and divides the data stream into individual data units 105. The storage controller 110 performs the process 400 (e.g. Figure 4A ) to calculate the maximum entropy value ("MAX") and the minimum entropy value ("MIN").

[0061] Instructions 640 may be executed to determine whether the received data unit is data reducible based on at least the minimum entropy value and the maximum entropy value. Instructions 650 may be executed to perform at least one reduction operation on the received data unit in response to determining that the received data unit is data reducible. For example, referring to Figure 1A , the storage controller 110 executes process 300 (eg Figure 3 ) to compare the maximum entropy value and the minimum entropy value with corresponding thresholds (e.g., LT1, UT1, LT2, UT2), and thereby determine that the data unit can be reduced. In response to this determination, the storage controller 110 performs at least one reduction operation (e.g., deduplication, compression) to attempt to reduce the received data unit. In some examples, if the comparison of the maximum entropy value and the minimum entropy value with the corresponding thresholds is insufficient to determine whether the data unit is feasible, the storage controller 110 may also determine the Shannon entropy value of the data unit and perform a reduction operation based on the comparison of the Shannon entropy value and the corresponding threshold (e.g., Figure 3 A comparison of the data units (as shown in blocks 360 and 370) is performed to determine whether the data unit can be reduced.

[0062] Figure 7 - Sample computing device

[0063] Figure 7 1 shows a schematic diagram of an example computing device 700. In some examples, computing device 700 may generally correspond to storage system 100 (e.g., Figure 1A As shown, computing device 700 may include a hardware processor 702 and a machine-readable storage device 705 containing instructions 710 to 750. Machine-readable storage device 705 may be a non-transitory medium. Instructions 710-750 may be executed by hardware processor 702 or by a processing engine included in hardware processor 702.

[0064] Instructions 710 may be executed to receive a data unit to be stored in a persistent storage device of a deduplication storage system. Instructions 720 may be executed to determine a minimum entropy value for the received data unit. Instructions 730 may be executed to determine a maximum entropy value for the received data unit. Instructions 740 may be executed to determine whether the received data unit is data reducible based on at least the minimum entropy value and the maximum entropy value. Instructions 750 may be executed to, in response to determining that the received data unit is data reducible, perform at least one reduction operation on the received data unit.

[0065] Figure 8 - Example process for storing data

[0066] Figure 8 An example process 800 for storing data according to some embodiments is shown. In some examples, the storage controller 110 ( Figure 1A ) to perform process 800. Process 800 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustration purposes, reference may be made to an example of an embodiment according to some embodiments. Figures 1A to 2B Details are described for process 800. However, other implementations are possible.

[0067] Block 810 may include receiving, by a storage controller of a storage system, a data unit to be stored in a persistent storage device of the storage system. Block 820 may include determining, by the storage controller, a minimum entropy value for the received data unit. Block 830 may include determining, by the storage controller, a maximum entropy value for the received data unit. Block 840 may include determining, by the storage controller, whether the received data unit is data reducible based on at least the minimum entropy value and the maximum entropy value. Block 850 may include, in response to determining that the received data unit is data reducible, performing, by the storage controller, at least one reduction operation on the received data unit.

[0068] According to some embodiments described herein, a storage system may include a storage controller for determining whether to attempt data reduction on a particular data unit. In some embodiments, the storage controller may analyze an incoming data unit to determine one or more entropy values ​​for the data unit, and may determine that the data unit is reducible when the entropy value satisfies a corresponding threshold. If it is determined that the data unit is reducible, the storage controller may attempt to reduce the data unit. In this way, the storage controller may avoid attempting to reduce data units that are not reducible. Thus, the storage controller may save processing time and / or energy that would otherwise be wasted in unproductive reduction attempts, and may thereby improve the performance and efficiency of the storage system.

[0069] Note that although Figures 1A to 8 Various examples are shown, but the embodiments are not limited in this respect. For example, referring to Figure 1A It is contemplated that the storage system 100 may include additional devices and / or components, fewer components, different components, a different arrangement, etc. In another example, it is contemplated that the functionality of the storage controller 110 described above may be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0070] Data and instructions are stored in corresponding storage devices implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

[0071] Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system having potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or product). An article or product may refer to any manufactured single component or multiple components. The one or more storage media may be located in the machine that runs the machine-readable instructions, or at a remote site from which the machine-readable instructions may be downloaded over a network for execution.

[0072] In the foregoing description, numerous details have been set forth to facilitate an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A storage system comprising: processor; as well as a machine-readable storage device storing instructions executable by the processor to: receiving a data unit to be stored in a persistent storage device of the storage system; determining a first entropy value for the received data unit; determining a second entropy value for the received data unit; Comparing the first entropy value with a first lower threshold; Comparing the second entropy value with a first upper threshold; as well as In response to at least one of determining that the first entropy value is not less than the first lower threshold and determining that the second entropy value is not less than the first upper threshold: Comparing the first entropy value with a second lower threshold; Comparing the second entropy value with a second upper threshold; as well as In response to at least one of determining that the first entropy value is not less than the second lower threshold and determining that the second entropy value is not less than the second upper threshold: storing the data unit in the persistent storage without attempting to reduce the data unit.

2. The storage system of claim 1 , comprising instructions executable by the processor to: In response to determining that the first entropy value is less than the first lower threshold and the second entropy value is less than the first upper threshold, at least one reduction operation is performed on the received data unit.

3. The storage system of claim 1 , comprising instructions executable by the processor to: In response to determining that the first entropy value is less than the second lower threshold, and the second entropy value is less than the second upper threshold: determining a Shannon entropy value of the received data unit; comparing the Shannon entropy value with a third threshold; as well as In response to determining that the Shannon entropy value is less than the third threshold, performing at least one reduction operation on the received data unit; as well as In response to determining that the Shannon entropy value is not less than the third threshold, the data unit is stored in the persistent storage without attempting to reduce the data unit.

4. The storage system of claim 1 , comprising instructions executable by the processor to: The first entropy value and the second entropy value are calculated using a lookup table, wherein: The lookup table stores a plurality of constants associated with a plurality of index values, wherein each constant represents a binary logarithm of a different index value.

5. The storage system according to claim 2 or 3, wherein: The at least one reduction operation is selected from a data deduplication operation and a data compression operation.

6. The storage system of claim 1 , comprising instructions executable by the processor to: calculating entropy values ​​of a plurality of received data units; determining whether a consecutive number of the reduced data units is greater than a first consecutive threshold; In response to determining that the consecutive number of reduced data units is greater than the first consecutive threshold: Pause the calculation of the entropy value of subsequent data units; attempting to perform data reduction on the subsequent data unit without performing the calculation of the entropy value; and In response to detecting a failed reduction of at least one data unit, the calculation of the entropy value is resumed.

7. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to: receiving a data unit to be stored in a persistent storage device of a storage system; determining a Rayleigh entropy value of the received data unit using a first technique; determining a Hartley entropy value for the received data unit using a second technique different from the first technique; determining whether the received data unit is capable of data reduction based at least on the Rayleigh entropy value of the received data unit and the Hartley entropy value of the received data unit; as well as In response to determining that the received data unit is capable of data reduction, at least one reduction operation is performed on the received data unit.

8. The non-transitory machine-readable medium of claim 7, comprising instructions that, when executed, cause the processor to: Comparing the Rayleigh entropy value with a first lower threshold; comparing the Hartley entropy value with a first upper threshold; and In response to determining that the Rayleigh entropy value is less than the first lower threshold and the Hartley entropy value is less than the first upper threshold, performing the at least one reduction operation on the received data unit.

9. The non-transitory machine-readable medium of claim 8, comprising instructions that, when executed, cause the processor to: In response to at least one of determining that the Rayleigh entropy value is not less than the first lower threshold value and determining that the Hartley entropy value is not less than the first upper threshold value: Comparing the Rayleigh entropy value with a second lower threshold; comparing the Hartley entropy value with a second upper threshold; as well as In response to at least one of determining that the Rayleigh entropy value is not less than the second lower threshold and determining that the Hartley entropy value is not less than the second upper threshold: storing the data unit in the persistent storage without attempting to reduce the data unit.

10. The non-transitory machine-readable medium of claim 9, comprising instructions that, when executed, cause the processor to: In response to determining that the Rayleigh entropy value is less than the second lower threshold value and the Hartley entropy value is less than the second upper threshold value: determining a Shannon entropy value of the received data unit; comparing the Shannon entropy value with a third threshold; as well as In response to determining that the Shannon entropy value is less than the third threshold, performing the at least one reduction operation on the received data unit.

11. The non-transitory machine-readable medium of claim 10, comprising instructions that, when executed, cause the processor to: In response to determining that the Shannon entropy value is not less than the third threshold, the data unit is stored in the persistent storage without attempting to reduce the data unit.

12. The non-transitory machine-readable medium of claim 7, comprising instructions that, when executed, cause the processor to: The Rayleigh entropy value and the Hartley entropy value are calculated using a lookup table, wherein, The lookup table stores a plurality of constants associated with a plurality of index values, wherein each constant represents a binary logarithm of a different index value.

13. The non-transitory machine-readable medium of claim 7, wherein: The at least one reduction operation is selected from a data deduplication operation and a data compression operation.

14. A method for data reduction, comprising: Receiving, by a storage controller of a storage system, a data unit to be stored in a persistent storage device of the storage system; determining, by the storage controller, a first entropy value for the received data unit using a first technique; determining, by the storage controller, a second entropy value for the received data unit using a second technique different from the first technique; determining, by the storage controller, whether the received data unit is capable of a data compression operation based on at least the first entropy value of the received data unit and the second entropy value of the received data unit; as well as In response to determining that the received data unit is capable of data reduction based on the first entropy value and the second entropy value, the storage controller performs the data compression operation on the received data unit.

15. The method of claim 14, comprising: Comparing the first entropy value with a first lower threshold; Comparing the second entropy value with a first upper threshold; as well as In response to determining that the first entropy value is less than the first lower threshold and the second entropy value is less than the first upper threshold, at least one reduction operation is performed on the received data unit.

16. The method of claim 15, comprising: In response to at least one of determining that the first entropy value is not less than the first lower threshold and determining that the second entropy value is not less than the first upper threshold: Comparing the first entropy value with a second lower threshold; Comparing the second entropy value with a second upper threshold; as well as In response to at least one of determining that the first entropy value is not less than the second lower threshold and determining that the second entropy value is not less than the second upper threshold: storing the data unit in the persistent storage without attempting to reduce the data unit.

17. The method of claim 16, comprising: In response to determining that the first entropy value is less than the second lower threshold, and the second entropy value is less than the second upper threshold: determining a Shannon entropy value of the received data unit; comparing the Shannon entropy value with a third threshold; as well as In response to determining that the Shannon entropy value is less than the third threshold, performing the data compression operation on the received data unit.

18. The method of claim 14, comprising: The first entropy value and the second entropy value are calculated using a lookup table, wherein the lookup table stores a plurality of constants associated with a plurality of index values, and wherein each constant represents a binary logarithm of a different index value.

19. The method of claim 14, wherein: The first entropy value is a minimum entropy value; and The second entropy value is a maximum entropy value.

20. The method of claim 19, wherein: The first entropy value is a Rayleigh entropy value; and The second entropy value is the Hartley entropy value.

Images