DATA REDUCTION IN A STORAGE SYSTEM

Abstract

A storage system (100) comprising: a processor; and a machine-readable memory that stores instructions, wherein the instructions can be executed by the processor to: receive a data unit (105) to be stored in a persistent memory (140) of the storage system; determine a first entropy value for the received data unit; determine a second entropy value for the received data unit; compare the first entropy value with a first lower threshold (LT1); compare the second entropy value with a first upper threshold (UT1); and in response to at least one determination that the first entropy value is not less than the first lower threshold, and a determination that the second entropy value is not less than the first upper threshold: compare the first entropy value with a second lower threshold (LT2);to compare the second entropy value with a second upper threshold (UT2); and in response to at least one determination that the first entropy value is not less than the second lower threshold, and a determination that the second entropy value is not less than the second upper threshold, to store the data unit in the persistent memory without attempting to reduce the data unit.

Description

background

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Some examples of data reduction techniques are data compression and data deduplication. Data deduplication identifies duplicate data units and attempts to reduce or eliminate the number of duplicate data units stored in the storage system. Brief description of the drawings

[0002] Some embodiments are described using the following figures. Fig. are schematic representations of an example memory system in accordance with some embodiments. Fig. is an illustration of an example lookup table in accordance with some implementations. Fig. is an illustration of a sample process in accordance with some implementations. Fig. are illustrations of example processes in accordance with some implementations. Fig. is an illustration of a sample process in accordance with some implementations. Fig. is a diagram of a machine-readable medium that stores instructions in accordance with some implementations. Fig. is a schematic diagram of an example computing device according to some implementations. Fig. is an illustration of a sample process in accordance with some implementations.

[0003] In the drawings, identical reference numbers indicate similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to clarify the example shown. Furthermore, the drawings contain examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description

[0004] In this disclosure, the use of the term "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Likewise, the term "comprises," "including," "comprises," "includes," "has," or "have," when used in this disclosure, specifies the presence of the specified elements but does not preclude the presence or addition of other elements.

[0005] In some examples, a storage system may perform data reduction operations to attempt to reduce the storage space required to store the received data. The data reduction operations may include data compression operations, data deduplication operations, or a combination thereof. For example, a compression operation may consist of encoding information using fewer bits than used in its original form. Further, a deduplication operation may include splitting an incoming data stream into multiple data units and determining which incoming data units are duplicates of previously stored data units. For an incoming data unit that is identified as a duplicate, the storage system may store a reference to the previous data unit instead of storing the duplicate incoming data unit.

[0006] In some examples, a deduplication operation may include comparing fingerprints of the incoming data units with fingerprints of the stored data units. As used herein, the term "fingerprint" refers to a value derived by applying a function to the contents of the data unit (where the "contents" may include all or a subset of the contents of the data unit). An example of a function that may be applied is a hash function that generates a hash value based on the incoming data unit. Examples of hash functions include cryptographic hash functions such as the Secure Hash Algorithm 2 (SHA-2) hash functions, e.g., SHA-224, SHA-256, SHA-384, etc. In other examples, other types of hash functions or other types of fingerprint functions may also be used.

[0007] A "storage system" may include a storage device or an array of storage devices. A storage system may also include one or more storage controllers that manage access to the storage device(s). A "unit of data" may refer to any piece of data that can be separately identified in the storage system. In some cases, a unit of data may refer to a chunk, a collection of chunks, or any other piece of data. In some examples, a storage system may store units of data in persistent storage. Persistent storage may be achieved using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or the like, or a combination thereof.

[0008] A "control device" may refer to a hardware processing circuit, which may include any one or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuit. Alternatively, a "control device" may refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry.

[0009] In some examples, a deduplication storage system may receive and store different data types (e.g., from different applications, with different types of content, etc.). However, some data types may not be suitable for reduction. Examples of data that are not suitable for reduction include data that does not contain repeating patterns, that is essentially random, and so on. Furthermore, attempting to reduce such data may consume relatively large amounts of processing time and / or energy, but provide little or no benefit in terms of reducing the storage space required to store the received data. Accordingly, attempting to reduce data that is not suitable for reduction may result in relatively slow and / or inefficient performance of the storage system.

[0010] In accordance with some implementations of the present disclosure, a memory system may include a memory controller to determine whether to attempt data reduction for particular data units. In some implementations, the memory controller may analyze an incoming data unit to determine one or more entropy values ​​for that data unit and may determine that the data unit is eligible for reduction if the entropy values ​​meet appropriate thresholds. If the data unit is determined to be eligible for reduction, the memory controller may attempt to reduce that data unit. In this way, the memory controller may avoid attempting to reduce data units that are not eligible for data reduction.Accordingly, the memory controller can save processing time and / or energy that would otherwise be wasted on unproductive reduction attempts, thereby improving the performance and efficiency of the memory system. FIG. 1A-1B - Example storage system

[0011] Fig. shows an example of a storage system 100 including a storage controller 110, a memory 115, and persistent storage 140, in accordance with some implementations. As illustrated, persistent storage 140 may include reduced data 150 and non-reduced data 160. Persistent storage 140 may include one or more non-transitory storage media, such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, etc., or a combination thereof. Memory 115 may be implemented as a semiconductor memory, e.g., random access memory (RAM).

[0012] In some implementations, memory controller 110 may include or execute data reduction logic 120 and filtering logic 130. Memory controller 110 and the included logic 120, 130 may be implemented as a processor that executes instructions (e.g., software and / or firmware) stored in a machine-readable storage medium, hardware (e.g., circuitry), etc.

[0013] In some implementations, the data reduction logic 120 may perform reduction operations to attempt to reduce selected data units 105. As in Fig. For example, as shown, data reduction logic 120 may include deduplication logic 122 and compression logic 126. Deduplication logic 122 may perform data deduplication operations, including generating a fingerprint for a received data unit 105 (e.g., a full or partial hash value) and comparing the generated fingerprint to stored fingerprints of previously received data units. If this comparison results in a match, it may be determined that a duplicate of the received data unit 105 is already stored in storage system 100. Further, in some implementations, compression logic 126 may perform data compression of data units. For example, compression logic 126 may perform one or more of run-length encoding, Huffman encoding, arithmetic encoding, Lempel-Ziv algorithms, etc.

[0014] In some implementations, filter logic 130 may determine whether a received data unit 105 is suitable for reduction, and if so, may cause data reduction logic 120 to attempt to reduce that data unit 105. For example, filter logic 130 may determine one or more numerical values ​​indicative of entropy characteristics (referred to herein as "entropy values") for a data unit 105 and may compare the entropy values ​​to corresponding thresholds 137 to determine whether the data unit 105 is suitable for reduction. If the comparison of the entropy values ​​to the corresponding thresholds 137 indicates that a data unit 105 is suitable for reduction, filter logic 130 may cause data reduction logic 120 to attempt to reduce the data unit 105.If the reduction is successful, the reduced data unit 105 may be stored in the persistent storage 140 as reduced data 150. However, if the comparison of the entropy values ​​with the thresholds 137 indicates that the data unit 105 is not suitable for reduction, no reduction of the data unit 105 is attempted, and the data unit 105 is stored in the persistent storage 140 as unreduced data 160. In this way, the filter logic 130 may reduce the amount of processing time and / or energy that would be wasted attempting to reduce irreducible data units 105, thereby improving the performance of the storage system 100.

[0015] In some examples, the reduction operations performed by data reduction logic 120 may be insufficient to reduce the size of a data unit 105 by at least a minimum amount or ratio (e.g., at least a 2:1 space reduction). Such reduction operations may be referred to herein as "failed" reductions. The minimum reduction required by storage system 100 may be specified in a system configuration setting. In some implementations, upon detecting a failed reduction of a data unit 105, data reduction logic 120 may store that data unit 105 in unreduced form in persistent storage 140 (e.g., as unreduced data 160).

[0016] In some implementations, filter logic 130 may use multiple entropy values, including a maximum entropy value, a minimum entropy value, and a Shannon entropy value. As used herein, the term "maximum entropy" may refer to a value that measures the maximum possible degree of uncertainty present in a set of data samples. Further, the term "minimum entropy" may refer to a value that measures the lowest possible degree of uncertainty in a group of data samples.

[0017] In some examples, a discrete random variable X can have n possible values, where the i-th outcome has the probability p iIn such examples, the maximum entropy (also called Hartley entropy) can be calculated as the binary logarithm of the number of values ​​that X can take with a non-zero probability, which can be represented as log2 n. In some examples, the minimum entropy (also called Rényi entropy) can be calculated as the negative binary logarithm of the probability of the most likely outcome, which can be represented as -log2 (max p i ). Furthermore, in some examples, the Shannon entropy can be calculated as the negative of the sum of the probability of each outcome multiplied by the binary logarithm of the probability of each outcome, which is written as -Σ n (p i * log2 (p i )) can be displayed.

[0018] In some implementations, the filter logic 130 may use modified calculations to calculate the maximum, minimum, and Shannon entropy values. For example, the modified calculations performed by the filter logic 130 may use integer arithmetic to calculate the maximum, minimum, and Shannon entropy values. Examples of the modified calculations performed by the filter logic 130 are described below with reference to the Fig. . In another example, the modified calculations use a lookup table 135 to calculate the maximum, minimum, and Shannon entropy values. The lookup table 135 may be a stored data structure containing an array of constants to determine binary logarithm values. An example implementation of the lookup table 135 is described below with reference to Fig. discussed. In some implementations, the modified calculations performed by filter logic 130 may avoid performing floating-point arithmetic as well as directly calculating binary logarithms. Accordingly, the use of the modified calculations may reduce the processing load associated with using filter logic 130.

[0019] In some implementations, filter logic 130 may periodically identify a data unit 105 that has an entropy value that is within a certain amount of the corresponding threshold 137 (e.g., does not meet threshold 137 but is within a certain distance of threshold 137). Filter logic 130 may attempt to reduce this identified data unit 105 and may determine whether the reduction attempt failed. If it determines that the reduction attempt did not fail, filter logic 130 may adjust threshold 137 such that the entropy value of data unit 105 meets the adjusted threshold 137.In this way, the filter logic 130 can periodically adjust the thresholds 137 to better match the entropy values ​​associated with the data units that are candidates for reduction and can thereby improve the performance of data reduction in the storage system 100. FIG. 2A-2B - Example data structures

[0020] Fig. shows an example of a lookup table 135 in accordance with some implementations. As shown, the lookup table 135 may contain multiple entries, where each entry corresponds to a different combination of an index value I and a constant value C. In some implementations, the filter logic 130 (shown in Fig. compare an input with the index values ​​I of the lookup table 135 and thereby determine the constant value C corresponding to the input value. It is noted that the index values ​​I in Fig. Although they are represented as data fields stored in the lookup table 135, other implementations are also possible. For example, the lookup table 135 can be implemented as an ordered array of constants, and the index value I for a particular constant C can be determined by the respective position of that constant C in the ordered array (e.g., the first position has the index I=0, the second position has the index I=1, etc.).

[0021] In some implementations, the index values ​​I of the lookup table 135 may correspond to a defined range of integer values ​​(e.g., 0 to 256). Further, for each index value I, the corresponding constant C may be a floating-point number of a specific length (e.g., seven digits) calculated as the binary logarithm of the index value I (i.e., log2(I)). Accordingly, in such implementations, the binary logarithm of I may be determined by looking up the input I in the lookup table 135. In this way, the use of the lookup table 135 may enable the determination of logarithm values ​​without using floating-point arithmetic, thus reducing the processing required by the filter logic 130 to determine entropy values.

[0022] Fig. shows an example implementation of a data structure in which the threshold values ​​137 are stored. In some implementations, the threshold values ​​137 may include, in ascending order, a first lower threshold (LT1), a second lower threshold (LT2), a third threshold (T3), a first upper threshold (UT1), and a second upper threshold (UT2). However, other implementations are also possible. The threshold values ​​137 may be compared to the corresponding entropy values ​​generated by the filter logic 130 to determine whether a particular data unit is eligible for data reduction. An example process for comparing the threshold values ​​137 to the corresponding entropy values ​​is described below with reference to Fig. discussed. FIG. 3 - Example process for storing data

[0023] Fig. shows an example process 300 for storing data in accordance with some implementations. In some examples, the process 300 may be executed using the memory controller 110 (shown in Fig. The process 300 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustrative purposes, the details of the method 300 are described below with reference to the Fig. described, which show examples of some implementations. However, other implementations are also possible.

[0024] Block 310 may include receiving a data unit to be stored in persistent storage. Block 320 may include determining a minimum entropy value (MIN) for the received data unit. Block 330 may include determining a maximum entropy value (MAX) for the received data unit. After block 330, the process 300 may continue at decision block 340 (see below). For example, the memory controller 110 (see Fig. an incoming data stream and divides the data stream into separate data units. The filter logic 130 performs modified calculations to determine maximum and minimum entropy values. In some implementations, these modified calculations are performed using the lookup table 135. An example process for calculating the maximum and minimum entropy values ​​is described below with reference to Fig. discussed.

[0025] Decision block 340 may include determining whether the minimum entropy value (MIN) is less than a first lower threshold (LT1) and the maximum entropy value (MAX) is less than a first upper threshold (UT1). Upon a positive determination ("YES") in decision block 340, process 300 may continue at block 380, including attempting data reduction of the data unit. Referring to Fig. For example, the memory controller 110 determines that the minimum entropy value (MIN) is less than the first lower threshold (e.g., LT1 equal to 4.0, as in Fig. shown), and also notes that the maximum entropy value (MAX) is less than the first upper threshold (e.g. UT1 equal to 6.0, as in Fig. shown). In response to these determinations, storage controller 110 performs one or more data reduction operations on data unit 105 (e.g., using deduplication logic 122 and / or compression logic 126). If the reduction of data unit 105 is successful, data unit 105 is stored as reduced data 150 in persistent storage 140.

[0026] However, upon a negative determination (“NO”) in decision block 340 (i.e., if the minimum entropy value (MIN) is not less than the first lower threshold (LT1) or if the maximum entropy value (MAX) is not less than the first upper threshold (UT1)), the process 300 may continue to decision block 350, including determining whether the minimum entropy value (MIN) is less than a second lower threshold (LT2) and the maximum entropy value (MAX) is less than a second upper threshold (UT2). Upon an affirmative determination (“YES”) in decision block 350, the process 300 may continue to block 360, including determining a Shannon entropy value (SE) for the received data unit. After block 360, the process 300 may continue to decision block 370 (see below). Upon a negative determination (“NO”) in decision block 350 (i.e.However, if the minimum entropy value (MIN) is not less than the second lower threshold (LT2) or if the maximum entropy value (MAX) is not less than the second upper threshold (UT2)), the process 300 may continue at block 390 (see below). Referring to FIG. Fig. For example, the memory controller 110 determines that the minimum entropy value (MIN) is less than the second lower threshold (e.g., LT2 equal to 5.0, as in Fig. shown), and also notes that the maximum entropy value (MAX) is less than the second upper threshold (e.g. UT2 equal to 7.5, as in Fig. shown). In response to these determinations, the filter logic 130 performs a modified calculation to determine a Shannon entropy value. In some embodiments, this modified calculation is performed using the lookup table 135. An example process for calculating the Shannon entropy value is described below with reference to Fig. discussed.

[0027] In decision block 370, it may be determined whether the Shannon entropy value (SE) is less than a third threshold (T3). If the determination is affirmative ("YES") in decision block 370, the process 300 may continue to block 380 (as described above). However, if the determination is negative ("NO") in decision block 370 (i.e., if the Shannon entropy value (SE) is not less than a third threshold (T3)), the process 300 may continue to block 390, including storing the data unit without performing data reduction. After either block 380 or block 390, the process 300 may be complete. In some examples, the process 300 may be repeated for multiple data units 105 received from the storage system 100. Fig. For example, if the memory controller 110 determines that the Shannon entropy value (SE) is less than the third threshold (e.g., T3 equals 5.5, as in Fig. shown), the storage controller 110 performs one or more data reduction operations on the data unit 105 (e.g., using the deduplication logic 122 and / or the compression logic 126). If the reduction of the data unit 105 is successful, the data unit 105 is stored in the persistent storage 140 as reduced data 150. Otherwise, if the Shannon entropy (SE) value is not less than the third threshold, the storage controller 110 does not reduce the data unit 105 and instead stores the data unit 105 as unreduced data 160 in the persistent storage 140. If the reduction of the data unit 105 is unsuccessful, the storage controller 110 stores the data unit 105 in the persistent storage 140 as unreduced data 160. FIG. 4A-4B - Example methods for calculating entropy values

[0028] Fig. shows an example method 400 for calculating the maximum and minimum entropy values. Furthermore, Fig. an example process 405 for calculating a Shannon entropy value. In some examples, the processes 400, 405 may be executed using the memory controller 110 (shown in Fig. The processes 400, 405 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, e.g., an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustrative purposes, details of the processes 400, 405 may be described below with reference to the Fig. described, which show examples of some implementations. However, other implementations are also possible.

[0029] Referring to Fig. Block 410 may include initializing a maximum probability variable (“Max_Prob”) to a value of 1. Block 420 may include initializing a cardinality variable (“Cardinality”) to a value of 0. Block 430 may include obtaining N samples from the data unit. For example, the memory controller 110 (see Fig. Generate samples of 8 randomly selected bytes, with each sample taken from each of the 32 evenly distributed blocks within a 16384-byte data unit (e.g., a cache page).

[0030] In block 440, a loop (defined by blocks 440-448) may be initiated to process each sample in the data unit (obtained in block 430). Block 442 may involve determining the value I of the current sample. Block 444 may involve determining a count of the sample value I (“count(I)”) (i.e., the number of times the value I has appeared so far in the samples of the data unit). If the count of the sample value I is currently equal to 0, the cardinality variable may be incremented by 1. In some implementations, the count of the sample value I (“count(I)”) may be determined using a histogram function.

[0031] In block 444, the count of sample value I may be incremented by 1. In block 448, if the count of sample value I is greater than the current maximum likelihood variable, the maximum likelihood variable may be set equal to the current count of sample value I. After block 448, the loop may return to block 440 (i.e., to process another sample in the data unit). After all samples have been processed in block 440, process 400 may continue in block 450.

[0032] In block 450, the maximum entropy value ("MAX") may be set equal to the value of the formula Int(A(B - log2(Max_Prob)) (i.e., the integer value of the product of constant A times the difference between constant B minus the binary logarithm of the maximum likelihood variable). For example, in some implementations, variable A may be equal to 10 and variable B may be equal to 8. Further, in some implementations, the binary logarithm of the maximum likelihood variable may be determined by looking up the maximum likelihood variable in lookup table 135.

[0033] In block 450, the minimum entropy value ("MIN") may be set equal to the value of the formula Int(A * log2 (MaxCardinality)) (i.e., the integer product of the constant A and the binary logarithm of the cardinality variable). In some implementations, the binary logarithm of the cardinality variable may be determined by looking up the cardinality variable in the lookup table 135. After block 460, the process 400 may be complete.

[0034] As in Fig. As shown, a loop (defined by blocks 470-475) may be entered in block 470 to process each sample value I (determined in block 442 in Fig. in the data unit. In block 475, the value of the total variable ("Total") may be set equal to the previous value of the total variable plus the value of the formula count(I)*(B - log(count(I))) (i.e., the product of the number of sample values ​​I times the difference between the variable B minus the binary logarithm of the number of sample values ​​I). In some embodiments, the binary logarithm of the count of the sample value I may be determined by looking up the count of the sample value I in the lookup table 135.

[0035] After all sample values ​​I have been processed in block 470, process 405 may continue in block 480. In block 480, the Shannon entropy value (SE) may be set equal to the value of the formula Int((A*Total)»B) (i.e., the integer value of the product of the constant A and the total variable, with the product bitwise right-shifted by B bits). After block 480, process 405 may be completed. FIG. 5 - Example process for storing data

[0036] Fig. shows an example process 500 for storing data in accordance with some implementations. In some examples, the process 500 may be executed using the memory controller 110 (shown in Fig. The process 500 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing machine, multiple processing machines, etc. For illustrative purposes, details of the method 500 are described below with reference to the Fig. described, which show examples of some implementations. However, other implementations are also possible.

[0037] Block 510 may include calculating entropy values ​​for received data units. Block 510 may include attempting data reduction for data units with entropy values ​​that satisfy entropy thresholds. For example, the memory controller 110 (see Fig. an incoming data stream and divides the data stream into separate data units 105. The memory controller 110 performs modified calculations to determine the entropy values ​​for a data unit 105 and compares the entropy values ​​to the corresponding threshold values ​​137 to determine whether the data unit 105 is eligible for reduction. If a data unit 105 is determined to be eligible for reduction, the memory controller 110 attempts to reduce that data unit 105. If the reduction of the data unit 105 is successful, the data unit 105 is stored as reduced data 150 in the persistent storage 140. Otherwise, if it is determined that a data unit 105 cannot be reduced, the memory controller 110 does not reduce that data unit 105 and instead stores it in the persistent storage 140 as unreduced data 160.If the reduction of the data unit 105 is unsuccessful, the memory controller 110 stores the data unit 105 as unreduced data 160 in the persistent memory 140.

[0038] In decision block 530, it may be determined whether the continuous number of reduced data units is greater than a first continuous threshold value ("CT1"). If the determination is negative ("NO") in decision block 530, the process 500 may continue in decision block 550 (see below). Otherwise, if the determination is positive ("YES") in decision block 530, the process 500 may continue in block 535, including interrupting the calculation of entropy values ​​for subsequent data units. Block 540 may include attempting data reduction of the subsequent data units without calculating their entropy values. Block 545 may include resuming the calculation of entropy values ​​for the subsequent data units in response to detecting a failed data reduction. After block 545, the process 500 may return to block 510. With respect to Fig. For example, the memory controller 110 determines that the consecutive number of reduced data units (i.e., the number of consecutive data units being reduced based on their entropy values) is greater than the CT1 threshold. The memory controller 110 then begins attempting to reduce each subsequent data unit 105 without calculating their entropy values. If a failed data reduction of at least one data unit 105 is detected (i.e., a data reduction in which the size of a data unit 105 cannot be reduced by at least a certain ratio), the memory controller 110 resumes calculating the entropy values ​​for subsequent data units 105 and uses the entropy values ​​to determine whether reduction operations should be performed on those data units 105.

[0039] At decision block 550, it may be determined whether the continuous number of unreduced data units is greater than a second continuous threshold ("CT2"). Upon a negative determination ("NO") at decision block 550, the process 500 may return to block 510 (i.e., continue calculating entropy values ​​for the received data units). Otherwise, upon a positive determination ("YES") at decision block 550, the process 500 may continue at block 560, including calculating entropy values ​​for a subset of the data units. Block 570 may include storing data units without attempting data reduction. Block 580 may include resuming the calculation of entropy values ​​for all data units in response to detecting data units with entropy values ​​that meet entropy thresholds. After block 580, the process 500 may return to block 510. With respect to Fig. For example, the memory controller 110 determines that the consecutive number of unreduced data units (i.e., the number of consecutive data units that are not reduced based on their entropy values) is greater than the CT2 threshold. The memory controller 110 then begins storing each subsequent data unit 105 without attempting to reduce those data units 105. Further, the memory controller 110 calculates entropy values ​​for a subset of the data units 105 (e.g., one of fifty data units) and compares the entropy values ​​to the thresholds 137 to determine whether a data unit 105 of the subset is eligible for reduction. If at least a certain number (e.g., 1, 5, etc.)) of data units 105 is determined to be viable, the memory controller 110 continues to calculate the entropy values ​​for subsequent data units 105 and uses the entropy values ​​to determine whether reduction operations should be performed for those subsequent data units 105.

[0040] In some implementations, process 500 may provide a feedback mechanism to decide whether to attempt data reduction based on information about the most recent number of reduced or unreduced data units. Accordingly, the application of data reduction may be automatically increased under conditions where data reduction is more effective. Furthermore, data reduction may be automatically decreased when data reduction is less effective. In this way, process 500 may improve the performance of the storage system. FIG. 6 - Example of a machine-readable data carrier

[0041] Fig. shows a machine-readable medium 600 on which instructions 610-650 are stored, in accordance with some implementations. The instructions 610-650 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 600 may be a non-transitory storage medium, such as an optical, semiconductor, or magnetic storage medium.

[0042] Instruction 610 may be executed to receive a data unit to be stored in the persistent storage of a deduplication storage system. Instruction 620 may be executed to determine a minimum entropy value for the received data unit. Instruction 630 may be executed to determine a maximum entropy value for the received data unit. For example, storage controller 110 (see Fig. an incoming data stream and divides the data stream into separate data units 105. The memory controller 110 executes the process 400 (shown in Fig. to calculate the maximum entropy value (“MAX”) and the minimum entropy value (“MIN”).

[0043] Instruction 640 may be executed to determine whether the received data unit is suitable for data reduction based on at least the minimum entropy value and the maximum entropy value. Instruction 650 may be executed to perform at least one reduction operation on the received data unit in response to determining that the received data unit is suitable for data reduction. For example, referring to Fig. , the memory controller 110 executes at least a portion of the process 300 (shown in Fig. ) to compare the maximum and minimum entropy values ​​with the corresponding thresholds (e.g., LT1, UT1, LT2, UT2), thereby determining that the data unit is suitable for data reduction. In response to this determination, the storage controller 110 performs at least one reduction operation (e.g., deduplication, compression) to attempt to reduce the received data unit. In some examples, if comparing the maximum and minimum entropy values ​​with the corresponding thresholds is not sufficient to determine whether the data unit is viable, the storage controller 110 may also determine a Shannon entropy value for the data unit and, based on a comparison of the Shannon entropy value with a corresponding threshold, determine whether the data unit is viable for reduction (e.g., as described in fields 360 and 370 in Fig. shown). FIG. 7 - Example data processing system

[0044] Fig. shows a schematic diagram of an example computing device 700. In some examples, the computing device 700 may generally correspond to a portion or all of the storage system 100 (shown in Fig. As illustrated, computing device 700 may include a hardware processor 702 and a machine-readable memory 705 with instructions 710-750. Machine-readable memory 705 may be a non-transferable medium. Instructions 710-750 may be executed by hardware processor 702 or by a processing engine included in hardware processor 702.

[0045] Instruction 710 may be executed to receive a data unit to be stored in persistent storage of a deduplication storage system. Instruction 720 may be executed to determine a minimum entropy value for the received data unit. Instruction 730 may be executed to determine a maximum entropy value for the received data unit. Instruction 740 may be executed to determine, based at least on the minimum entropy value and the maximum entropy value, whether the received data unit is suitable for data reduction. Instruction 750 may be executed to perform at least one reduction operation on the received data unit in response to determining that the received data unit is suitable for data reduction. FIG. 8 - Example process for storing data

[0046] Fig. shows an example process 800 for storing data in accordance with some implementations. In some examples, the process 800 may be performed using the memory controller 110 (shown in Fig. The process 800 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. For illustrative purposes, details of the process 800 are described below with reference to the Fig. described, which show examples of some implementations. However, other implementations are also possible.

[0047] Block 810 may include receipt by a storage controller of a storage system of a data unit to be stored in persistent storage of the storage system. Block 820 may include determination by the storage controller of a minimum entropy value for the received data unit. Block 830 may include determination by the storage controller of a maximum entropy value for the received data unit. Block 840 may include the storage controller determining, based on at least the minimum entropy value and the maximum entropy value, whether the received data unit is suitable for data reduction. Block 850 may include, in response to determining that the received data unit is suitable for data reduction, the storage controller performing at least one reduction operation on the received data unit.

[0048] In accordance with some implementations described herein, a storage system may include a storage controller to determine whether to attempt data reduction for particular data units. In some implementations, the storage controller may analyze an incoming data unit to determine one or more entropy values ​​for that data unit and may determine that the data unit is eligible for reduction if the entropy values ​​meet appropriate thresholds. If the data unit is determined to be eligible for reduction, the storage controller may attempt to reduce that data unit. In this way, the storage controller may avoid attempting to reduce data units that are not eligible for data reduction.Accordingly, the memory controller can save processing time and / or energy that would otherwise be wasted on unproductive reduction attempts, thereby improving the performance and efficiency of the memory system.

[0049] Please note that the Fig. As various examples show, implementations are not limited in this respect. For example, the storage system 100 as shown in Fig. shown, additional devices and / or components, fewer components, different components, different arrangements, etc. As another example, the functionality of the storage controller 110 described above may be included in a different machine or software of the storage system 100. Other combinations and / or variations are also possible.

[0050] Data and instructions are stored in corresponding storage devices embodied as one or more computer-readable or machine-readable storage media. Storage media includes various forms of non-transitory memory, including semiconductor memory such as dynamic or static random-access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs), and flash memory; magnetic disks such as fixed, floppy, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs), or other types of storage devices.

[0051] It should be noted that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed throughout a large system, possibly with multiple nodes. Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture may refer to each individual component manufactured, or to multiple components. The storage medium or media may be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions may be downloaded over a network for execution.

[0052] In the foregoing description, numerous details are set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details described above. The appended claims are intended to cover such modifications and variations.

Claims

[1] A storage system comprising: a processor; and a machine-readable memory that stores instructions, the instructions being executable by the processor to: receive a unit of data to be stored in a persistent memory of the storage system; determine a minimum entropy value for the received data unit; determine a maximum entropy value for the received data unit; Determining, based on at least the minimum entropy value and the maximum entropy value, whether the received data unit is suitable for data reduction; and in response to determining that the received data unit is suitable for data reduction, perform at least one reduction operation on the received data unit. [2] The memory system of claim 1, including instructions executable by the processor to: compare the minimum entropy value with a first lower threshold; compare the value of the maximum entropy with a first upper threshold; and in response to determining that the minimum entropy value is less than the first lower threshold and the maximum entropy value is less than the first upper threshold, performing at least one reduction operation on the received data unit. [3] The memory system of claim 2, including instructions executable by the processor to: in response to determining that the minimum entropy value is not less than the first lower threshold and / or determining that the maximum entropy value is not less than the first upper threshold: compare the minimum entropy value with a second lower threshold; compare the value of the maximum entropy with a second upper threshold; and in response to at least a determination that the minimum entropy value is not less than the second lower threshold and a determination that the maximum entropy value is not less than the second upper threshold, store the data unit in the persistent storage without attempting to reduce the data unit. [4] The memory system of claim 3, including instructions executable by the processor to: in response to the determination that the minimum entropy value is less than the second lower threshold and the maximum entropy value is less than the second upper threshold: Determining a Shannon entropy value for the received data unit; compare the Shannon entropy value with a third threshold; and in response to determining that the Shannon entropy value is less than the third threshold, performing at least one reduction operation on the received data unit. [5] The memory system of claim 4, including instructions executable by the processor to: in response to determining that the Shannon entropy value is not less than the third threshold, store the data unit in the persistent storage without attempting to reduce the data unit. [6] The memory system of claim 1, including instructions executable by the processor to: Calculating the minimum entropy value and the maximum entropy value using a lookup table, wherein the lookup table stores a plurality of constants associated with a plurality of index values, and wherein each constant represents a binary logarithm of a different index value. [7] The storage system of claim 1, wherein the at least one reduction operation is selected from a data deduplication operation and a data compression operation. [8] The memory system of claim 1, including instructions executable by the processor to: to calculate entropy values ​​for a plurality of received data units; determine whether a consecutive number of reduced data units is greater than a first consecutive threshold; in response to determining that the continuous number of reduced data units is greater than the first continuous threshold: suspend calculation of entropy values ​​for subsequent data units; Attempt to reduce the subsequent data units without calculating entropy values; and in response to the detection of a failed reduction of at least one data unit, to resume the calculation of the entropy values. [9] A non-transitory machine-readable medium that stores instructions that, when executed, cause a processor to: receive a unit of data to be stored in a persistent store of a storage system; determine a minimum entropy value for the received data unit; determine a maximum entropy value for the received data unit; Determining, based on at least the minimum entropy value and the maximum entropy value, whether the received data unit is suitable for data reduction; and in response to determining that the received data unit is suitable for data reduction, perform at least one reduction operation on the received data unit. [10] The non-transitory machine-readable medium of claim 9, including instructions that, when executed, cause the processor to: compare the minimum entropy value with a first lower threshold; compare the value of the maximum entropy with a first upper threshold; and in response to determining that the minimum entropy value is less than the first lower threshold and the maximum entropy value is less than the first upper threshold, performing at least one reduction operation on the received data unit. [11] The non-transitory machine-readable medium of claim 10, including instructions that, when executed, cause the processor to: in response to determining that the minimum entropy value is not less than the first lower threshold and / or determining that the maximum entropy value is not less than the first upper threshold: compare the minimum entropy value with a second lower threshold; compare the value of the maximum entropy with a second upper threshold; and in response to at least a determination that the minimum entropy value is not less than the second lower threshold and a determination that the maximum entropy value is not less than the second upper threshold, store the data unit in the persistent storage without attempting to reduce the data unit. [12] The non-transitory machine-readable medium of claim 11, including instructions that, when executed, cause the processor to: in response to the determination that the minimum entropy value is less than the second lower threshold and the maximum entropy value is less than the second upper threshold: Determining a Shannon entropy value for the received data unit; compare the Shannon entropy value with a third threshold; and in response to determining that the Shannon entropy value is less than the third threshold, performing at least one reduction operation on the received data unit. [13] The non-transitory machine-readable medium of claim 12, including instructions that, when executed, cause the processor to: in response to determining that the Shannon entropy value is not less than the third threshold, store the data unit in the persistent storage without attempting to reduce the data unit. [14] The non-transitory machine-readable medium of claim 9, including instructions that, when executed, cause the processor to: Calculating the minimum entropy value and the maximum entropy value using a lookup table, wherein the lookup table stores a plurality of constants associated with a plurality of index values, and wherein each constant represents a binary logarithm of a different index value. [15] The non-transitory machine-readable medium of claim 9, wherein the at least one reduction operation is selected from a data deduplication operation and a data compression operation. [16] A procedure comprising: Receiving, by a storage controller of a storage system, a data unit to be stored in a persistent memory of the storage system; Determination of a minimum entropy value for the received data unit by the memory controller; Determination of a maximum entropy value for the received data unit by the memory controller; Determining, by the memory controller, based on at least the minimum entropy value and the maximum entropy value, whether the received data unit is suitable for data reduction; and in response to determining that the received data unit is suitable for data reduction, performing at least one reduction operation on the received data unit by the memory controller. [17] The method of claim 16, comprising: Comparing the minimum entropy value with a first lower threshold; Comparing the maximum entropy value with a first upper threshold; and in response to determining that the minimum entropy value is less than the first lower threshold and the maximum entropy value is less than the first upper threshold, performing the at least one reduction operation on the received data unit. [18] The method of claim 17, comprising: in response to determining that the minimum entropy value is not less than the first lower threshold and / or determining that the maximum entropy value is not less than the first upper threshold: Comparison of the minimum entropy value with a second lower threshold; Comparing the maximum entropy value with a second upper threshold; and in response to at least a determination that the minimum entropy value is not less than the second lower threshold and a determination that the maximum entropy value is not less than the second upper threshold, storing the data unit in the persistent storage without attempting to reduce the data unit. [19] The method of claim 18, comprising: in response to the determination that the minimum entropy value is less than the second lower threshold and the maximum entropy value is less than the second upper threshold: Determining a Shannon entropy value for the received data unit; Comparing the Shannon entropy value with a third threshold; and in response to determining that the Shannon entropy value is less than the third threshold, performing the at least one reduction operation on the received data unit. [20] The method of claim 16, comprising: Calculating the minimum entropy value and the maximum entropy value using a lookup table, wherein the lookup table stores a plurality of constants associated with a plurality of index values, and wherein each constant represents a binary logarithm of a different index value.

Images