Storage system

Abstract

To improve a compression rate of a data.SOLUTION: A controller performs an inline compression of plaintext data from a host and a post-process compression of inline compressed data stored in one or more storage drives. In the inline compression, a data compression / decompression circuit performs compression processing including a first dictionary compression on the plaintext data from the host to generate the inline compressed data, and stores the inline compressed data in one or more storage drives. In the post-process compression, the inline compressed data read from the one or more storage drives is decompressed by the data compression / decompression circuit to generate plaintext data. In the post-process compression, the compression processing including a second dictionary compression by a processor having a character string search capability superior to that of the first dictionary compression is performed on the plaintext data to generate the post-process compressed data, and the post-process compressed data is stored in the one or more storage drives.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a data compression technique in a storage system. [Background technology]

[0002] Storage systems are information devices used to store and manage large amounts of data, and the cost per unit of capacity can be reduced by storing more data. For this reason, some storage systems are equipped with a function to compress written data before storing it on the disk drive.

[0003] For example, in recent years, solid-state drives (SSDs) equipped with NAND flash memory, a nonvolatile semiconductor memory, have been adopted as storage media in storage systems in addition to or instead of hard disk drives (HDDs). Because SSDs do not have a physical head-seek mechanism for data access like HDDs, they have less latency and superior response performance for random data reads.

[0004] For this reason, in applications such as databases that require high-speed random reads, SSDs are increasingly replacing HDDs as the storage media in storage systems. However, although the bit cost of SSDs has been decreasing year by year due to the increasing integration of flash memory cells, it still remains higher than the bit cost of HDDs.

[0005] Therefore, many storage systems that use SSDs as storage media have adopted lossless compression technology to reduce the size of data stored on the SSD, which makes the system's storage capacity appear virtually larger, reducing the cost per capacity and bringing it closer to that of storage systems that use HDDs as storage media.

[0006] Patent Document 1 is a background technology of the present disclosure. Patent Document 1 discloses a storage system that uses the LZMA algorithm. In LZMA compression processing, plaintext data before compression is first subjected to dictionary compression processing. The dictionary compression result is then subjected to range encoding processing, thereby generating LZMA compressed data. In LZMA decompression processing, the compressed data is first subjected to range decoding processing. The decoded result is then subjected to plaintext expansion processing, thereby generating the original plaintext data. Patent Document 1 speeds up inline compression and decompression by configuring the LZMA range code to have multiple ranges and processing the range coder-decoder in parallel. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2022-095015 Summary of the Invention [Problem to be solved by the invention]

[0008] Inline compression, which compresses data received from the host and stores it on the storage drive, is required to have a high compression ratio and meet the IO performance requirements of the storage system. Data compression using hardware circuits (accelerators) enables high-speed data compression and can meet higher IO performance requirements. However, storage system design imposes limitations on the logical size of the accelerator's FPGA (Field Programmable Gate Array). Therefore, there is a need for technology that improves the compression rate of stored data while satisfying the IO performance requirements of storage systems and the limitations on the circuit scale for data compression. [Means for solving the problem]

[0009] A storage system according to one embodiment of the present invention includes a controller including a processor and a data compression and decompression circuit, wherein the controller performs inline compression of plaintext data from a host and post-process compression of the inline compressed data stored in one or more storage drives, wherein the inline compression includes performing a compression process including first dictionary compression on the plaintext data from the host by the data compression and decompression circuit to generate inline compressed data and storing the inline compressed data in the one or more storage drives, and wherein the post-process compression includes decompressing the inline compressed data read from the one or more storage drives by the data compression and decompression circuit to generate the plaintext data, and performing a compression process including second dictionary compression by the processor, which has better string search capabilities than the first dictionary compression, on the plaintext data to generate post-process compressed data and storing the one or more storage drives. [Effects of the Invention]

[0010] According to one aspect of the present invention, data stored in a storage system can be compressed more effectively. [Brief explanation of the drawings]

[0011] [Figure 1] 1 shows an example of the configuration of a storage system. [Figure 2A] This shows an overview of the LZMA algorithm. [Figure 2B] A specific example of dictionary compression processing will be shown. [Figure 2C] An example of the configuration of a hash table is shown below. [Figure 3A] FIG. 1 shows a functional block diagram of range encoding. [Figure 3B] FIG. 10 shows a functional block diagram of range decoding. [Figure 4A] An example will be given to explain the principle of range coding. [Figure 4B] Another example will be given to explain the principle of range coding. [Figure 5A] 10 shows a flowchart of a range encoding process. [Figure 5B] 10 shows a flowchart of a range decoding process. [Figure 6] FIG. 1 shows a functional block diagram of a method for increasing the speed of range encoding processing. [Figure 7] 10 shows a flowchart of a method for increasing the speed of range encoding processing. [Figure 8] FIG. 1 shows a functional block diagram of a method for increasing the speed of range decoding processing. [Figure 9] 10 shows a flowchart of a method for increasing the speed of range decoding processing. [Figure 10] FIG. 10 is a logical configuration diagram for explaining post-process compression. [Figure 11] 10 is a flowchart illustrating post-process compression. [Figure 12] FIG. 10 is a logical configuration diagram for explaining another example of post-process compression. [Figure 13] 10 is a flowchart illustrating another example of post-process compression. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following, when necessary for convenience, the description will be divided into multiple sections or examples, but unless otherwise specified, they are not unrelated to each other, and one is related to the other as a partial or complete modification, detail, supplementary explanation, etc. Furthermore, in the following, when the number of elements, etc. (including the number, numerical value, amount, range, etc.) is mentioned, it is not limited to that specific number, and may be more or less than the specific number, unless otherwise specified or when it is clearly limited in principle to a specific number, etc.

[0013] In one embodiment of this specification, write data to a storage system is compressed in an inline process by an accelerator (logic circuit) and written to a storage drive. Furthermore, the inline compressed data written to the storage drive is converted back to plain text in a post-process, and then recompressed by the accelerator using a compression method that strengthens dictionary compression and written back to the storage drive. This makes it possible to improve the data compression rate of the storage system (further reducing the size of stored data) without increasing the circuit scale of the accelerator.

[0014] LZMA compression can improve the compression ratio by increasing the search capability of dictionary compression. However, LZMA compression using hardware circuits (accelerators) increases the logic scale of the FPGA (Field Programmable Gate Array) by increasing the search capability of dictionary compression. Furthermore, software dictionary compression in inline compression can cause a decrease in IO performance.

[0015] It is difficult to improve the inline compression ratio by strengthening dictionary compression due to the IO performance requirements of the storage system and the FPGA logic scale of the accelerator. Therefore, a technology is needed to improve the compression ratio of stored data while satisfying the IO performance requirements of the storage system and the circuit scale limitations for data compression. [Example]

[0016] (1) System configuration In the following, a storage system with a data compression function will be described as an embodiment of this specification, which reduces the amount of stored data by lossless compression.

[0017] 1 shows an example of the configuration of a storage system according to an embodiment of this specification. The storage system 101 includes a host I / F (Interface) 102, a storage controller 103, multiple SSDs (Solid State Drives) 105, and a cache memory 106 that uses volatile memory such as DRAM (Dynamic Random Access Memory).

[0018] The storage controller 103 is connected to the host I / F 102, SSD 105, and cache memory 106, and includes a CPU 107, which is a processor that controls them, and memory 109. The CPU 107 can include one or more cores, and operates as a predetermined functional unit by executing a program (software) stored in the memory 109.

[0019] The memory 109 stores system software including an operating system required to run programs on the storage controller 103, programs running on the processor 107, metadata used by the programs, and data that needs to be temporarily stored. Note that only one of the memory 109 and the cache memory 106 may be implemented, and data from the other may also be stored.

[0020] The CPU 107 interprets the contents of read / write commands from a host (not shown), transmits and receives data to and from the host, compresses and decompresses data using an LZMA compression / decompression circuit 104, which is a data compression / decompression circuit, and transfers data between the SSD 105 and the cache memory 106. The CPU 107 also performs post-process compression of data stored in the SSD 105.

[0021] The host I / F 102 is an interface mechanism for connecting to an external host, and responds to read / write commands in order to send data to the host and receive data from the host. The mechanism of the host I / F 102 and the protocol for sending and receiving commands and data comply with, for example, a standard interface specification.

[0022] The storage controller 103 includes an LZMA compression / decompression circuit 104 and a transfer circuit 108. The transfer circuit 108 receives and transmits data compressed or decompressed by the LZMA compression / decompression circuit 104. The transfer circuit 108 transfers data between components of the storage system 101, for example, between the LZMA compression / decompression circuit 104 and the cache memory 106, and between the CPU 107 and the cache memory 106. The transfer circuit 108 also transfers data between the host I / F 102 and the cache memory 106, and between the SSD 105 and the cache memory 106.

[0023] The LZMA compression / decompression circuit 104 reversibly compresses write data received in response to a write command to generate compressed data in order to reduce the amount of data stored in the SSD 105, which is a storage drive. Also, in order to transmit the original plaintext data to the host in response to a read command, the LZMA compression / decompression circuit 104 decompresses the compressed data read from the SSD 105 to generate plaintext data.

[0024] The storage controller 103 executes inline compression and post-process compression. Inline compression compresses write data from the host and stores it in the SSD 105. Post-process compression reads compressed data from the SSD 105, re-compresses the decompressed data, and returns it to the SSD 105.

[0025] In inline compression, write data from the host is first temporarily stored in the cache memory 106. At this point, the storage controller 103 returns a write completion signal to the host. The data is then converted into compressed data via the LZMA compression / decompression circuit 104, and the compressed data is also temporarily stored in the cache memory 106. The compressed data is then written to the SSD 105. As will be described later, the LZMA compression / decompression circuit 104 performs dictionary compression and range encoding.

[0026] The post-process compression reads the compressed data from the SSD 105 and stores it in the cache memory 106. Next, the post-process compression decompresses the compressed data using the LZMA compression / decompression circuit 104 to restore it to plain text. After that, the post-process compression re-compresses the data using dictionary compression, which has stronger string search capabilities than the LZMA compression / decompression circuit 104, and range encoding using the CPU 107. The compressed data is temporarily stored in the cache memory 106 and then written to the SSD 105.

[0027] Read data to the host is read in a compressed state from the SSD 105 and first temporarily stored in the cache memory 106. It is then converted into plaintext data through the LZMA compression / decompression circuit 104, and the plaintext data is also temporarily stored in the cache memory 106. The plaintext data is then transmitted to the host.

[0028] The LZMA compression / decompression circuit 104 is implemented as hardware (logic circuitry) designed based on the data compression / decompression method according to an embodiment of this specification, rather than as a processor that operates using software. For example, it may be an accelerator using an FPGA. The LZMA compression / decompression circuit 104 has high-speed data decompression performance, allowing the storage system 101 to take advantage of the high-speed random read performance that is a feature of SSDs, not only for uncompressed data but also for compressed data.

[0029] A storage drive of a type different from an SSD, for example, an HDD (Hard Disk Drive), may be used. The storage drive may not be installed inside the housing of the storage system 101, but may be connected to the storage controller 103 or cache memory 106 via a network, and the storage area of ​​the storage drive may exist on a cloud.

[0030] (2) LZMA algorithm As background knowledge for explaining the data decompression method according to an embodiment of this specification, the LZMA algorithm will be explained with reference to FIGS. 2A to 5B.

[0031] (2-1) Overview of the LZMA algorithm 2A shows an overview of the LZMA algorithm. In the LZMA compression process, plaintext data 201 before compression is first subjected to dictionary compression 202. The dictionary compression result is then subjected to range encoding 203. This generates LZMA compressed data 204.

[0032] On the other hand, in the LZMA decompression process, the compressed data 204 is first subjected to a range decoding process 205. The decoded result is then subjected to a plaintext expansion process 206. This generates the original plaintext data 201.

[0033] (2-2) Dictionary compression processing 2B shows a specific example of dictionary compression 202 that constitutes the LZMA algorithm. A check is made sequentially to see if the same character string reappears in the character string stream of plaintext data 201. If a character string matches L characters in a row, starting from the first character and ending with J characters before, this character string is converted into a copy symbol [L,J].

[0034] For example, in a four-character string 211 of "b,c,d,e," four consecutive characters match starting from six characters before the first character "b." In this case, the string 211 is converted to a copy symbol [4,6]. Similarly, in a four-character string 212 of "a,b,a,b," four consecutive characters match starting from two characters before the first character "a" (including overlapping parts). In this case, the string 212 is converted to a copy symbol [4,2].

[0035] Similarly, in the six-character string 213 "c,d,e,f,e,b," six consecutive characters match starting from the first character "c" and 15 characters before. In this case, the string 213 is converted to a copy symbol [6,15]. Since the amount of data of these copy symbols is less than the amount of data contained in the original string, this conversion can reduce the amount of data.

[0036] The range of the string stream (hereafter referred to as the dictionary) referenced in the match search is from one character before to a specified number of characters before. Because the dictionary range slides backward with each search, this compression technique is also called sliding dictionary compression. If there are multiple matching strings within the dictionary range, the longest consecutive matching string is converted into a copy symbol. This has the effect of further reducing the amount of data.

[0037] To generate data to be input to the subsequent range encoding 203, characters that have not been converted into copy symbols (hereinafter referred to as literal characters) and copy symbols must be encoded using a specified bit pattern and then concatenated to form a bit stream.

[0038] FIG. 2B shows the bit stream resulting from encoding according to the rules of the LZMA specification. This bit stream is input to range encoding 203. For example, bit pattern 221 is 12 bits long and represents the copy symbol [4,6]. Bit pattern 222 is 11 bits long and represents the copy symbol [4,2]. Bit pattern 223 is 13 bits long and represents the copy symbol [6,15]. In this way, the length of the bit pattern corresponding to the copy symbol is not fixed. On the other hand, a literal character is represented by a 9-bit bit pattern, which is obtained by adding one zero bit to the beginning of the 8-bit value of the character.

[0039] The range decoding process 205 outputs such a bit stream in the LZMA decompression process. When the plaintext expansion process 206 receives such a bit stream, it interprets it as copy symbols or literal characters and restores the character string stream of the plaintext data 201.

[0040] 2C shows an example of the configuration of a hash table 230 that the LZMA compression / decompression circuit 104 uses to search for matching character strings in the dictionary compression 202. The hash table 230 shows the three-character character strings from which each hash value is obtained and the position of those character strings in the input data. In the example shown in FIG. 2C, each hash value is represented by three bits, and the character string is three consecutive characters. The upper limit of the number of entries for each hash value is three.

[0041] The processing procedure of dictionary compression 202 will be explained below. As shown in Fig. 2B, the plaintext data that is input data has the following structure: "a, b, c, d, e, f, e, b, c, d, e, a, b, a, b, a, b, c, d, e, f, e, b, ...". The LZMA compression / decompression circuit 104 performs the following processes in order from the head position of the input data.

[0042] [1] First, the LZMA compression / decompression circuit 104 calculates a hash value X (3 bits) of three characters from the current position, and registers the current position and the three characters in the hash table 230. Note that due to the characteristics of hash functions, the same hash value may be calculated from completely different character strings.

[0043] [2] Next, if the number of registered entries for hash value X exceeds three, the LZMA compression / decompression circuit 104 discards the oldest entry (the entry with the smallest position value). If the character string appears frequently, information is registered in the entry with high frequency, so there is a high probability that an entry with the same character string will be found in the processing of step [3] below. On the other hand, character strings that appear infrequently have a low priority in terms of compression efficiency obtained by converting to copy codes, so the hash table can be used efficiently by preferentially excluding such character strings from the entries.

[0044] [3] Next, the LZMA compression / decompression circuit 104 refers to other entries related to hash value X to check whether there is an entry storing a string that matches the newly registered current string. If the strings stored in one or more entries for hash value X are the same, the LZMA compression / decompression circuit 104 selects the string that is closest to the current string. This selection operation can be understood as follows, assuming that abc is once again given as the current string in FIG. 2C. The new current string abc is registered in entry 3, and entry 2, which has a string that matches this current string, is identified. On the other hand, if no string matching the current string exists in the corresponding entry even after hash value X is calculated, the process proceeds to step [7].

[0045] [4] Next, since there is a possibility that four or more characters will match, the LZMA compression / decompression circuit 104 directly reads the input data and checks the longest matching length L. That is, the current string has a matching string in the entry linked to the corresponding hash value X, and the range of matching characters following that string is also checked. As a result, identical strings (three or more characters) that start with the string stored in the entry are identified in two positions, and the length of this identified string is calculated as the longest matching length L.

[0046] [5] Next, the LZMA compression / decompression circuit 104 determines the distance D (equal to the difference between the positions stored in the two compared entries) from the current position (the position of the current string) to the beginning of the found matching string.

[0047] [6] Next, the LZMA compression / decompression circuit 104 converts the character string of length L from the current position into a copy code [L, D], and proceeds to step [8].

[0048] [7] If there is no character string that matches the current character string in step [3], the LZMA compression / decompression circuit 104 converts the one character at the current position into a literal code.

[0049] [8] After step [6], if the end of the data has been reached, the LZMA compression / decompression circuit 104 ends the process. If not, the LZMA compression / decompression circuit 104 returns to step [3].

[0050] The dictionary code for the above plaintext data is as follows: "a,b,c,d,e,f,e,[4,6],a,b,[4,2],[6,15],...".

[0051] As described above, hash table 230 can be used to efficiently find matching three-character strings, and then matching strings of more characters can be found by checking for matches of four or more characters.

[0052] Generally, the larger the number of bits in the hash value (the number of rows in the hash table 230) and the larger the upper limit on the number of entries (the number of columns in the hash table 230), the more strings registered in the hash table 230 increase as a power of 2, improving the probability of finding a matching string. In other words, the compression rate of dictionary compression improves. However, improving the compression rate requires a large capacity of table memory, which increases the circuit size of the LZMA compression / decompression circuit 104. The probability of finding a matching string can also be improved by reducing the number of characters in a string. However, this increases the time required to search for a matching string in plaintext data. Note that the probability of finding a matching string can also be improved by increasing the number of entries allowed for a hash value. In other words, even if the occurrence frequency is low, the chances of finding a matching string can be increased by extending the period for which the string and its position are retained in the entry. However, adopting this method also increases the time required to search for a matching string.

[0053] (2-3) Range encoding and decoding process FIG. 3A shows a functional block diagram of range encoding, and FIG. 3B shows a functional block diagram of range decoding. The range encoding function 300 will be described with reference to FIG. 3A. The encoder 301 is a calculation block that receives input bit by bit from an input bit string 302 and generates an output code 303. In the example of FIG. 3A, the input bit string is "1,1,0,1", which corresponds to a part of the bit string shown in FIG. 2B. The method for generating the output code 303 will be described in the next section using FIGS. 4A and 4B. The output code 303 corresponds to compressed data of the LZMA algorithm (LZMA204 compressed data in FIG. 2A).

[0054] The encoder 301 also uses a probability value referenced from a probability table 304 as input. The probability value P(x) indicates the probability that the next input bit from the input bit string 302 is a "0" when the previously input bit history 305 is x. While the figure uses a typical notation of probability using a decimal point for simplicity, in practice, a sufficiently large integer range is given and the range is divided into integer widths (divisions) according to the probability. If the integer range becomes too small for further division, the number of digits is increased at that stage to allow further division, thereby avoiding problems such as floating points. Alternatively, the probability value P(x) may indicate the probability that the next input bit from the input bit string 302 is a "1" when the previously input bit history 305 is x.

[0055] The encoder 301 adapts the probability value P(x) through learning each time it uses it. For example, if the next input bit is actually "0", it increases P(x), and if it is "1", it decreases P(x). Note that at the start of encoding, all unused P(x) values ​​are 0.5 (the probability of "0" and "1" is equal).

[0056] For example, in Figure 3A, when the first bit is given, x = "-". The probability that the first bit is "0" is 0.4, and the probability that it is "1" is 0.6. If the bit string is given immediately after the encoder is fully initialized, P(-) = 0.5 (initial value). However, in Figure 3A, bit history learning is applied, resulting in P "-" being different from the initial value.

[0057] Also, when x=0, when the first bit is "0", the probability that the second bit will be "0" is 0.35, and the probability that it will be "1" is 0.65. In this case, too, if the second bit is "0", the probability of P(x=0) is increased, and if the second bit is "1", the probability of P(x=0) is decreased. These probabilities are updated through learning over the entire bit history (bit stream) in question. The compression mechanism achieved by applying the probability table 304 obtained in this way to the bit history 305 is explained in Figure 4A.

[0058] Next, the range decoding function 310 will be described with reference to Fig. 3B. The decoder 311 is an operation block that receives an input code 312 (i.e., an output code 303) and generates an output bit string 313 bit by bit. The generation method will be described in the next section with reference to Figs. 4A and 4B. The output bit string 313 corresponds to the input bit stream to the plaintext expansion process of the LZMA algorithm (plaintext expansion process 206 in Fig. 2), i.e., the output bit stream from the dictionary compression 202 in Fig. 2.

[0059] The decoder 311 also uses as input a probability value taken from a probability table 314. The probability value P(x) indicates the probability that the next output bit will be "0" when the bit history 315 output so far is x.

[0060] Like the encoder 301, the decoder 311 adapts the probability value P(x) through learning each time it uses it. For example, if the next output bit is actually "0", it increases P(x), and if it is "1", it decreases P(x). Note that at the start of decoding, all unused P(x) values ​​are 0.5 (the probability of "0" and "1" is equal).

[0061] When the output code 303 of the range encoding function 300 and the input code 312 of the range decoding function 310 are the same, the learning-induced changes of all probability values ​​P(x) in the probability tables 304 and 314 are the same. Therefore, the changes during encoding are reproduced during decoding.

[0062] (2-4) Principles of range encoding and decoding 4A and 4B show examples for explaining the principles of range encoding and decoding. The encoding process performed by the encoder 301 in FIG. 3A divides the numerical axis (0, 1) into lengths corresponding to the probability that each bit value of the input bit string is "0," and repeatedly leaves one of the sections as the division target for the next bit. The encoding process outputs the coordinate values ​​included in the final remaining section as a code. Note that if the bit is "0," the left-hand section of the division is left, and if the bit is "1," the right-hand section is left. The probability that each bit value is "0" is obtained from the probability table 304 using the input bit history up to that point as an index.

[0063] According to the LZMA algorithm, the bit history for referencing probability values ​​is cleared under certain conditions. For example, in the case of 9 bits representing a literal character, the encoder 301 encodes the first bit of the 8 bits excluding the 1 header bit using a null bit history. The encoder 301 encodes the last 8th bit, with bits 1 to 7 as the bit history, and then clears the bit history.

[0064] As shown in FIGS. 4A and 4B, the decoding process performed by the decoder 311 in FIG. 3B divides the [0, 1) numerical axis into segments with lengths corresponding to the probability that each bit value in the output bit string is "0." Furthermore, the decoder 311 checks which segment the input code (coordinate value) is included in, determines each bit in the output bit string, and leaves the segment that included it as a target for division for the next bit. Finally, all values ​​in the output bit string are determined.

[0065] When the division is performed, if the left section contains a code, the decoder 311 determines that the bit is "0", and if the right section contains a code, the decoder 311 determines that the bit is "1". The probability that each bit value is "0" is obtained from the probability table 314 using the output bit history up to that point as an index.

[0066] The decoder 311 clears the bit history for referencing the probability value under the same conditions as for encoding. For example, in the case of 9 bits representing a literal character, the decoder 311 decodes the 8 bits excluding the 1 header bit, with the 1st bit being a null bit history. For the last 8th bit, the decoder 311 decodes bits 1 to 7 as the bit history, and then clears the bit history.

[0067] 4A and 4B both show examples of transitions in numerical axis division during encoding and decoding of the bit string "1101." However, the probabilities of the bit value being "0" are different. In the example of FIG. 4A, the probability value of "0" is always 0.5. In the example of FIG. 4B, the probabilities of "0" are 0.25, 0.25, 0.75, and 0.25, respectively, in bit order. FIG. 4A shows a case where the probability values ​​referenced from the probability tables 304 and 314 are in the initial state.

[0068] Therefore, if the probability table 304 has already been trained, the intervals are divided according to the probabilities stored in the trained probability table 304. Furthermore, by inputting the bit string "1101," in FIG. 4A, the "1" sections in the first, second, and fourth columns are made larger than the "0" sections, and in the third column, the "0" sections are made larger than the "1" sections. The size of the "0" sections in each row (the value that defines the boundary between "1" and "0" when the left edge of each row is considered to be 0) is then registered in the probability table 304. (Note that when the bit string "1101" is encoded, P(x) in the x="-", "1", "10", and "110" rows is updated.) FIG. 4B shows a case where the probability values ​​referenced from the probability tables 304 and 314 have changed due to training.

[0069] In range encoding and decoding, if the bit string "1101" is frequently processed, it will learn that "1" is likely to appear first in probability tables 304 and 314, that if the history is "1", then "1" is likely to appear next, that if the history is "11", then "0" is likely to appear next, and that if the history is "110", then "1" is likely to appear next. As this learning of probabilities progresses and predictions of how bits will appear become more accurate, the output code of range encoding will become shorter.

[0070] The encoding flow in FIG. 4A is as follows. · First step: According to the input "1", the right half section [1 / 2~2 / 2] remains. · Second step: According to the input "1", the right half section [3 / 4~4 / 4] remains. · Third step: According to the input "0", the left half section [6 / 8~7 / 8] remains. ·Fourth step: According to the input "1", the right half section [13 / 16~14 / 16] remains. The output code for the last section is 13 / 16 (1101 in binary).

[0071] On the other hand, the coding flow in Figure 4B is as follows: · First step: According to the input "1", the right 3 / 4 section [1 / 4~4 / 4] remains. · Second step: According to the input "1", the right 3 / 4 section [7 / 16~16 / 16] remains. · Third step: According to the input "0", the left 3 / 4 section [28 / 64~55 / 64] remains. ·Fourth step: According to the input "1", the right 3 / 4 section [139 / 256~220 / 256] remains. The output code for the last section is 3 / 4 (11 in binary).

[0072] The more the bit string is input according to the probability prediction (i.e., the more it is identical to the bit history processed in the past), the larger the size of the interval remaining after division. Therefore, fewer bits are required to represent the coordinate values ​​of the output code included in the final remaining interval. In the example of Figure 4A, four bits are required because the probability table is the initial value, while in the example of Figure 4B, only two bits are required as a result of applying the learned probability table. In this way, range codes improve compression ratios by learning the bit occurrence probability according to the bit history.

[0073] The output code 401 and output code 411 obtained as a result of the encoding described above are provided as input codes in the decoding process. For example, when the output code 401 is provided as the input code, first, P(x) for x="-" is obtained from the probability table 314. Then, it is determined whether 13 / 16 falls within the "0" or "1" section for the probability of P(x="-"). In the case of FIG. 4A, P(x) is set to 0.5 for all x, so P(x="-") is calculated as 1 / 2, and 13 / 16 is determined to be within the 1 / 2 to 2 / 2 range of the evenly divided 0 / 2 to 2 / 2 section. As a result, the first bit is identified as "1."

[0074] Next, since the first bit has been identified, the probability table 314 is referenced and P(x="1") = 0.5 is obtained, and it is determined whether the 13 / 16 of the output code 401 falls within the "0" or "1" section. In this case, since 13 / 16 falls within the range of 3 / 4 to 4 / 4 in the section of 2 / 4 to 4 / 4, the second bit is identified as "1". Next, the probability table is referenced using the bit history (x=11), and the section containing the output code 401 is again found and the value of the corresponding bit is identified. By sequentially repeating this operation, it is possible to decode the input bit string from the output code.

[0075] The probability for each bit history (x) in probability table 314 is updated each time one bit is decoded, and the update follows the same process as when encoder 301 generates output code 303. That is, when input code 312 is decoded from the beginning (i.e., identified from the first bit of bit history 315), in the initial stage, section division is performed based on probability information close to the initial value, just as in the initial stage of generating output code 301 by encoder 301, and bit history 315 containing input code 312 is identified. If output code 303 and input code 312 are the same, the bit history identified when dividing the section will also be the same.

[0076] As decoding progresses, the probability table 314 is updated more and more, and probability information appears according to the characteristics of the bit history 315. The section division corresponding to the probability information learned according to the characteristics of this bit history 315 is the same as the section division during encoding. Therefore, if the input code 312 and the output code 303 are the same, the bit string specified for the calculated section division is uniquely specified, and it becomes possible to decode the input bit string 302 as the output bit string 313.

[0077] (2-5) Range encoding and decoding flowchart Fig. 5A shows a flowchart of an example of range encoding. Fig. 5B shows a flowchart of an example of range decoding. First, the procedure of range encoding will be described with reference to Fig. 5A.

[0078] The encoder 301 references the probability value of the next bit being "0" from the probability table 304 according to the input bit history (501). The encoder 301 divides the numerical axis range (the range to be divided) into two intervals according to the probability value (502). When dividing, the range size is multiplied by the probability value, which is the most time-consuming part of the encoding process. The encoder 301 then selects one of the two intervals according to whether the input bit value is "0" or "1" (503).

[0079] Next, in step 504, the encoder 301 determines whether the input of bits has ended. If the input of bits has ended (504: YES), the process proceeds to step 506; if there is still input (504: NO), the process proceeds to step 505.

[0080] In step 505, the encoder 301 updates the probability value used in the probability table 304 and updates the bit history for encoding the next bit. The probability value is updated by increasing it if the input bit value is "0" and decreasing it if it is "1." The bit history is updated by, for example, changing the input bit next to "11" to "110" if it is "0." After this, the encoder 301 returns to step 501 to continue the encoding process.

[0081] On the other hand, in step 506, the encoder 301 outputs the coordinate value that identifies the last remaining interval, for example, the value included in the interval that has the smallest number of bits for expression, as a code, and ends the encoding process.

[0082] An example of range decoding will be described with reference to FIG. 5B. The decoder 311 references the probability table 314 according to the output bit history to obtain the probability value that the next bit is "0" (511). The decoder 311 divides the numerical axis range into two intervals according to the probability value (512). When dividing, the range size is multiplied by the probability value. This is the most time-consuming part of the decoding process. The decoder 311 then selects one of the two intervals that contains the value of the input code (513). The decoder 311 outputs the bit value "0" or "1" represented by the selected interval (514).

[0083] Next, in step 515, the decoder 311 determines whether the output of bits has ended. If so (515: YES), the decoding process ends. If there is more output (515: NO), the process proceeds to step 516.

[0084] In step 516, the decoder 311 updates the probability value used in the probability table 314 and updates the bit history for decoding the next bit. The probability value is updated by increasing it if the output bit value is "0" and decreasing it if it is "1." The bit history is updated by changing it to "110," for example, if the output bit after "11" is "0." After this, the decoder 311 returns to step 511 to continue the decoding process.

[0085] The above coding is done by encoding and decoding a given bit string ( The probability table is updated each time one bit of the symbol stream is processed, and the longer the bit string being processed, the more appropriate a probability distribution is obtained for that bit string, resulting in higher compression efficiency (i.e., the later the bit string is located, the more likely it is to produce a high compression effect).

[0086] In addition to this method of adaptively updating the probability table, it is also possible to check all bit strings before encoding begins, create a probability table, and use this probability table as a fixed parameter for encoding and decoding. Compared to the adaptive method described above, when a probability table is created and used independently of the encoding and decoding processes, the probability (section division) is independent of the position within the bitstream being processed, making it possible to encode and decode from positions other than the beginning. In this case, however, the encoder 301 and decoder 311 must share the same probability table. (3) Accelerating range encoding

[0087] Figure 6 shows a functional block diagram of an example of a method for speeding up range encoding processing. The LZMA compression / decompression circuit 104 in Figure 1 performs the compression processing shown in this block diagram. In the example described with reference to Figures 3A to 5B, a multiplication process is performed by referencing a probability each time one bit is input. As a result, only one bit can be processed in one calculation cycle, which can cause the processing performance of the LZMA algorithm to be slow.

[0088] The range encoding function 600 shown in Figure 6 speeds up range encoding processing by operating multiple encoders simultaneously. That is, N ranges to be divided (N>1) are prepared, and N types of bit histories are prepared in advance from the input bit string. This range encoding function 600 simultaneously references N probability values ​​in a probability table and performs multiplication processing on the N input bits in parallel, improving encoding performance by N times.

[0089] Figure 6 shows an example of encoding using this high-speed technology when N=4. All four encoders 601A-601D perform the same processing as encoder 301 in Figure 3A. Each encoder retrieves and uses one probability value from probability table 604. These four probability values ​​are referenced using bit histories 605A-605D as indexes.

[0090] Bit history 605A is used when encoder 601A processes the first bit "1" of input bit string 602, and its value is null. Bit history 605B is used when encoder 601B processes the second bit "1" of input bit string 602, and its value is "1."

[0091] Bit history 605C is used when encoder 601C processes the third bit “0” of input bit string 602, and its value is “11.” Bit history 605D is used when encoder 601D processes the fourth bit “1” of input bit string 602, and its value is “110.”

[0092] In general, the bit history used to encode the Nth bit is formed by concatenating the 1st to (N-1)th bits. By preparing four types of bit histories in this way, the four encoders 601A to 601D can simultaneously refer to four probability values ​​from the probability table 604 and simultaneously perform multiplication using these probability values.

[0093] The four sub-codes 603A to 603D output from the encoders 601A to 601D are finally concatenated to form the output code 606. The output code 606 corresponds to the compressed data of the LZMA algorithm. According to this method, 4-bit input can be processed in one calculation cycle, improving the range encoding performance in the compression process of the LZMA algorithm by four times compared to conventional methods.

[0094] FIG. 7 shows an example of a flowchart of the range encoding acceleration method described with reference to FIG. 6. The procedure of the range encoding acceleration method will be described with reference to the flowchart in FIG. 7. First, the LZMA compression / decompression circuit 104 creates N types of bit histories used to encode N bits of the input bit string (701). N is an integer equal to or greater than 2. The N encoders obtain, from the probability table, the probability value that the next bit is "0" based on each bit history (702). The N encoders divide each of the N numerical axis ranges (ranges to be divided) into two intervals according to these probability values ​​(703).

[0095] The multiplication of the range size and the probability value by the N encoders is performed in parallel. In the first cycle, the numerical axis range (range size) is common to the N encoders, and is [0, 1] in the example shown in FIG. 6. In the second cycle and thereafter, the range selected by the relevant encoder in the previous cycle becomes the target numerical axis range (range size). Each encoder selects one of the two ranges on the left and right depending on whether the input bit value is "0" or "1" (704).

[0096] Next, in step 705, the LZMA compression / decompression circuit 104 determines whether the bit input is complete, and if the bit input is complete (705: YES), the process proceeds to step 707, and if there is more input (705: NO), the process proceeds to step 706. In step 706, the LZMA compression / decompression circuit 104 updates the N probability values ​​used in the probability table. The probability value is updated by increasing it if the input bit value is "0" and decreasing it if it is "1."

[0097] After this, the LZMA compression / decompression circuit 104 returns to step 701 to continue the encoding process. For example, if N is 4 and 8 bits of a literal character are to be encoded, the first 4 bits are encoded in the first cycle of this flow, and the last 4 bits are encoded in the second cycle. In the second cycle, the bit history used to encode the 5th bit is the bit string of the first 4 bits.

[0098] For example, if the input bit string is 6 bits, the first cycle may encode the first 4 or 3 bits, and the second cycle may encode the last 2 or 3 bits. The maximum value of the input bit string to the LZMA compression / decompression circuit 104 is 4, but bit strings below this value can be encoded.

[0099] In step 707, each encoder generates a coordinate value that identifies the last remaining interval, for example, the value contained in that interval that has the fewest number of bits to represent. The LZMA compression / decompression circuit 104 outputs a bit string that concatenates these N values ​​as a code, thereby completing the encoding process.

[0100] In the range encoding example shown in Figure 6, the integer N>1 and the number of input bits is N bits, i.e., the bit history length is at most (N-1) bits. This range encoding can speed up encoding by N times using N ranges to be divided. As mentioned in the explanation with reference to Figure 7 for the example of 8-bit literal characters, if the integer M>N and the number of input bits is M bits, i.e., the bit history length is at most (M-1) bits, then the N ranges to be divided can be used to speed up encoding of the input bits.

[0101] The method will be explained below using an example where M=8 and N=4. The LZMA compression / decompression circuit 104 has a 255-entry probability table indexed by a maximum 7-bit bit history. The LZMA compression / decompression circuit 104 prepares four types of bit history (empty, 1 bit, 2 bits, and 3 bits, respectively) to be used in encoding the first 4 bits of an 8-bit input bit string, and simultaneously references the four corresponding probability values ​​in the probability table. The LZMA compression / decompression circuit 104 uses these probability values ​​to encode the first 4 bits in parallel in the first cycle.

[0102] Next, the LZMA compression / decompression circuit 104 prepares four types of bit histories (4 bits, 5 bits, 6 bits, and 7 bits, respectively, including the first 4 bits at the beginning) to be used for encoding the last 4 bits of the input bits, and simultaneously references the corresponding four probability values ​​from the probability table.The LZMA compression / decompression circuit 104 uses these probability values ​​to encode the last 4 bits in parallel in the second cycle.

[0103] In this way, the LZMA compander 104 processes an 8-bit input in two cycles (i.e., four times the performance), generates four sub-codes, and constructs an output code by concatenating them.

[0104] In general, the performance of the encoding process for an M-bit input range code can be improved by using a probability table with (2^M-1) entries indexed by a bit history of up to (M-1) bits and N encoders to process M bits in [M / N] calculation cycles. Note that the LZMA compression / decompression circuit 104 may generate subcodes without performing the above parallel processing.

[0105] (4) Accelerating range decoding An N-fold increase in the speed of range decoding processing cannot be achieved by simply operating N decoders 311 in Figure 3B in parallel. This is because the bit history used in the processing of a certain decoder X is uncertain until decoder Y, which decodes the previous bit, outputs the processing result. Therefore, decoder X cannot look up probability values ​​from the probability table and perform multiplication using those values ​​simultaneously with decoder Y, and parallelization cannot be achieved.

[0106] A method for speeding up range decoding processing according to one embodiment of this specification will be described below. Fig. 8 shows a functional block diagram of the method for speeding up range decoding processing. The LZMA compression / decompression circuit 104 in Fig. 1 performs decompression processing using a range decoding function 800 shown in this block diagram.

[0107] FIG. 8 shows an example of range decoding when N=4, that is, the number of bits in the output bitstream is 4. All of the 15 decoders, 8A (1), 8B0 and 8B1 (2), 8C00 to 8C11 (4), and 8D000 to 8D111 (8), perform processing similar to that of the decoder 311 in FIG. 3B. Some of the decoders are not shown in FIG. 8. The four sub-codes 803A to 803D input to these 15 decoders are separated from the input code 802 (corresponding to compressed data in the LZMA algorithm) and are the same as the four sub-codes 603A to 603D in FIG. 6. Note that one sub-code may be shared by multiple decoders.

[0108] Each decoder uses one probability value from the probability table 804 to output a candidate bit value, with the 15 probability values ​​indexed into the full set of possible bit histories.

[0109] The bit history value used by one decoder 8A to decode the first bit of the output bit string 806 is NULL, and the bit history values ​​used by two decoders 8B0 and 8B1 to decode the second bit of the output bit string 806 are "0" and "1", respectively.

[0110] The bit history values ​​used by the four decoders 8C00 to 8C11 to decode the third bit of the output bit string 806 are "00," "01," "10," and "11," respectively. The bit history values ​​used by the eight decoders 8D000 to 8D111 to decode the fourth bit of the output bit string 806 are "000," "001," "010," "011," "100," "101," "110," and "111," respectively.

[0111] In general, the number of bit histories used to decode the K-th bit is 2^(K-1). Each bit history is a (K-1)-bit bit pattern (bit string) that is possible as the first through (K-1)-th bits of the output bit string 806. By preparing 15 types of bit histories in this way, 15 probability values ​​are simultaneously referenced from the probability table 804, and the 15 decoders simultaneously perform multiplication using these probability values.

[0112] If the first bit of the output bit string 806 output by the decoder 8A is "1", it is clear that the second bit output by the decoder 8B1, which performed decoding assuming that the first bit was "1" out of the two decoders 8B0 and 8B1, is the correct result. Therefore, the selector 805B selects "1" output by the decoder 8B1 from the two second bit candidates output by the decoders 8B0 and 8B1. In other words, the first and second bits are determined to be "11".

[0113] This shows that the third bit output by decoder 8C11, which performed decoding on the assumption that the first and second bits were "11" among decoders 8C00 to 8C11, is the correct result. Therefore, selector 805C selects "0" output by decoder 8C11 from the four third bit candidates output by decoders 8C00 to 8C11. In other words, the first to third bits are determined to be "110".

[0114] This shows that the fourth bit output by decoder 8D110, which performed decoding on the assumption that the first to third bits were "110" among decoders 8D000 to 8D111, is the correct result. Therefore, selector 805D selects "1" output by decoder 8D110 from the eight fourth bit candidates output by decoders 8D000 to 8D111.

[0115] As a result of the above, the four bits of the output bit string 806 are determined to be "1101." Generally speaking, the LZMA compression / decompression circuit 104 is equipped with 2^(K-1) decoders for decoding the Kth bit, and stores 2^(K-1) candidates for the Kth bit output by these decoders. The LZMA compression / decompression circuit 104 selects, as the Kth bit, a candidate output by one decoder that performed decoding assuming that the already determined values ​​of the 1st to (K-1)th bits are bit history.

[0116] The bit selection process by selectors 805B to 805D takes a much shorter time than the multiplication process by the decoder. Therefore, this method allows 4-bit output to be processed in one calculation cycle. As a result, the performance of the range decoding process in the decompression process of the LZMA algorithm is improved by four times compared to the conventional method.

[0117] The procedure for the high-speed range decoding method described with reference to Fig. 8 will be described below with reference to Fig. 9. First, the LZMA compression / decompression circuit 104 creates (2^N-1) bit histories that may be used to decode N bits of the output bit string (901). There are 2^(K-1) bit histories used to decode the Kth bit.

[0118] Each of the (2^N-1) decoders obtains a probability value indicating that the next bit is "0" from the probability table 804 according to the bit history it is responsible for (902), and divides the numerical axis range (range to be divided) into two intervals according to the probability value (903). The numerical axis range divided by the 2^(K-1) decoders used to decode the K-th bit is common. Specifically, the numerical axis range of all decoders in the first cycle is common, which is [0,1] in the example of Figure 8. From the second cycle onwards, the numerical axis range of the decoder for the K-th bit is the range of the division result by the decoder that output the correct answer for the K-th bit in the previous cycle. When dividing, the range size is multiplied by the probability value.

[0119] The decoder selects one of the two intervals that contains the input subcode value (904), and generates a bit value "0" or "1" represented by the selected interval (905). The number of bit values ​​generated is (2^N-1), and the number of candidates for the Kth bit is 2^(K-1). The selector then selects one correct bit from each candidate in order, starting from the first bit, and determines and outputs an N-bit pattern (906). The correct values ​​of the 1st to (K-1)th bits are used as bit history to select the correct value for the Kth bit.

[0120] Next, in step 906, the LZMA compression / decompression circuit 104 determines whether the output of bits has ended. If so (906: YES), the decoding process ends. If there is more output (906: NO), the process proceeds to step 907.

[0121] In step 907, the LZMA compression / decompression circuit 104 updates the N probability values ​​used for the probability table 804. The probability value is updated by increasing it if the output bit value is "0" and decreasing it if it is "1." Furthermore, the LZMA compression / decompression circuit 104 adopts the interval selected in step 904 by the decoder that output the correct bit value as the numerical axis range for the next cycle. The interval selected in step 904 by one decoder that output the correct value for the Kth bit out of the 2^(K-1) decoders for the Kth bit is adopted as the numerical axis range to be divided in step 903 for the next decoding of the Kth bit.

[0122] After this, the LZMA compression / decompression circuit 104 returns to step 901 to continue the decoding process. For example, if the first 4 bits and the last 4 bits of an 8-bit literal character are encoded in two cycles, the bit history used to decode the 5th bit in the second cycle of this flow is the bit string of the first 4 bits.

[0123] For example, if 6-bit input data is divided into the first 4 bits and the remaining 2 bits and coded in two cycles, the LZMA compression / decompression circuit 104 may decode 4 or 3 bits in the first cycle, and then decode 2 or 3 bits in the second cycle. The maximum value of the input bit string to the LZMA compression / decompression circuit 104 is 4, and bit strings of less than this value can be decoded.

[0124] In the method for accelerating the decoding process of a range code shown in Fig. 8, when the integer N>1, the number of input bits is N bits (i.e., the length of the bit history is at most (N-1) bits), and N ranges to be divided are used, thereby speeding up the decoding process by N times. As mentioned with reference to Fig. 9 for the example of an 8-bit literal character, when the integer M>N, the number of input bits is M bits, i.e., the length of the bit history is at most (M-1) bits, N ranges to be divided can be used to speed up the decoding of the output bit string.

[0125] An example of decoding an 8-bit output bit string will be described below. The LZMA compression / decompression circuit 104 has 15 decoders, as in Figure 8, and inputs four sub-codes separated from the input code (corresponding to the compressed data of the LZMA algorithm) to these 15 decoders, as in Figure 8. Each decoder obtains and uses one probability value from a 255-entry probability table, indexed by a maximum 7-bit bit history.

[0126] The 15 probability values ​​referenced in the first cycle are values ​​referenced using as indices all possible bit histories (null, 1 bit, 2 bits, 3 bits, respectively) for the first 4 bits of the 8-bit output bit string. The bit history value used by one decoder to decode the first bit of the output bit string is null.

[0127] The two decoders that decode the second bit of the output bitstream use bit history values ​​of "0" and "1", respectively. The four decoders that decode the third bit of the output bitstream use bit history values ​​of "00", "01", "10", and "11", respectively. The eight decoders that decode the fourth bit of the output bitstream use bit history values ​​of "000", "001", "010", "011", "100", "101", "110", and "111", respectively.

[0128] The 15 decoders perform multiplication in parallel using the probability values ​​referenced in their bit history. Then, as in Figure 8, the selector's bit selection process determines the values ​​of the first through fourth bits in order. In this example, we will use "1101."

[0129] Next, the 15 probability values ​​referenced in the second cycle are values ​​referenced using as indexes all possible bit histories for the last 4 bits of the 8-bit output bit string (4 bits, 5 bits, 6 bits, and 7 bits, respectively, starting with "1101" determined in the first cycle).

[0130] The bit history value used by one decoder to decode the fifth bit of the output bitstream is "1101." The bit history values ​​used by two decoders to decode the sixth bit of the output bitstream are "11010" and "11011," respectively. The bit history values ​​used by four decoders to decode the seventh bit of the output bitstream are "110100," "110101," "110110," and "110111," respectively.

[0131] The bit history values ​​used by the eight decoders to decode the eighth bit of the output bit string are "1101000", "1101001", "1101010", "1101011", "1101100", "1101101", "1101110", and "1101111", respectively.

[0132] The 15 decoders perform multiplication in parallel using the probability values ​​referenced in their bit histories. Then, as in Figure 8, the selector's bit selection process determines the values ​​of bits 5 to 8 in order.

[0133] As in Figure 8, the selector's bit selection process is performed in a much shorter time than the decoder's multiplication process. Therefore, this method allows 8-bit output to be processed in two calculation cycles. As mentioned above, in a probability table with 255 entries, 15 entries are referenced in the first cycle, and 15 entries are selected and referenced from the remaining 240 entries in the second cycle. In the second cycle, the number of entries referenced is reduced to 1 / 16 by indexing using the bit history including the first 4 bits determined in the first cycle.

[0134] In general, the decoding process for a range code that outputs M bits can be improved by using a probability table with (2^M-1) entries indexed by a bit history of up to (M-1) bits and (2^N-1) decoders to process M bits in [M / N] calculation cycles.

[0135] As described above, one embodiment of this specification can quickly decompress data compressed using range coding, thereby improving the read response performance of compressed data in, for example, an appliance storage system that has a data compression function using a range coding algorithm.

[0136] (5) Post-processing compression Post-process compression is explained below. Post-process compression can improve the compression rate of data stored in SSD 105 while suppressing the impact on write access from the host to the storage system. Fig. 10 is a logical configuration diagram for explaining post-process compression, and Fig. 11 is a flowchart for explaining post-process compression.

[0137] 10 , inline compression involves compressing write data 211 from the host using the LZMA compression / decompression circuit 104 and storing the inline-compressed data in the SSD 105. Post-process compression involves decompressing the inline-compressed data stored in the SSD 105 using the LZMA compression / decompression circuit 104, recompressing the decompressed data using the CPU 107, and storing the decompressed data in the SSD 105.

[0138] After dictionary compression 171, CPU 107 performs range encoding 172 that is compatible with range encoding 203 in LZMA compression / decompression circuit 104 (uses the same compression / decompression algorithm). Dictionary compression 171 (second dictionary compression) by CPU 107 has a higher search capability for matching character strings than dictionary compression 202 (first dictionary compression) by LZMA compression / decompression circuit 104. This makes it possible to increase the compression rate of post-process compression compared to that of in-line compression. Furthermore, because range encoding 172 by CPU 107 is compatible with range encoding 203 in LZMA compression / decompression circuit 104, decompression can be performed using LZMA compression / decompression circuit 104 in read processing, thereby suppressing a decrease in read performance.

[0139] Note that the range encoding 203 and 172 may be omitted, and encoding using another compression / decompression algorithm may be performed. The range encoding may be performed by a process different from the parallel processing described above. The data compression rate can be increased by performing dictionary compression and encoding different from dictionary compression, such as entropy encoding such as range coding or Huffman coding.

[0140] 11, in step 1001, CPU 107 compares the current operating rate of CPU 107 with a preset threshold and determines whether the operating rate is less than the threshold. If the operating rate is equal to or greater than the threshold (1001: NO), this flow ends. If the operating rate is less than the threshold (1001: YES), this flow proceeds to step 1002.

[0141] By starting post-process compression when the load on the CPU 107 is less than a threshold, it is possible to reduce the impact on other processes. A value that represents the CPU load different from the CPU utilization rate, such as the number of running tasks, may be referenced. The CPU load does not have to be referenced, and post-process compression may be performed periodically, for example.

[0142] For example, the CPU 107 may perform post-process compression together with garbage collection. The CPU 107 adds updated data at a certain address in the volume to a new address on the SSD 105. The old data at the address stored on the SSD 105 becomes invalid data. The garbage collection collectively stores valid data in the SSD 105 in a new address area, and changes the area of ​​invalid data to free space. When moving valid data, the CPU 107 performs post-process compression on the target data read from the SSD 105 and stores it at a new address on the SSD 105. This allows post-process compression to be performed efficiently.

[0143] In step 1002, the CPU 107 selects and reads one piece of inline compressed data from the SSD 105, and stores it in the cache memory 106. As described above, inline compression compresses write data from the host using the LZMA compression / decompression circuit 104 and stores it in the SSD 105.

[0144] Data stored in SSD 105 and not yet subjected to post-process compression may be managed by management information (not shown). For example, the management information may include an address within the volume, information on data that has not yet been subjected to post-process compression, and address information of the SDD 105 that stores valid data at that address. The management information may be stored in memory within storage controller 103.

[0145] The management information may include information on whether or not valid stored data has undergone post-process compression, as well as information on the time the data was stored (updated). CPU 107 may select data for which post-process compression is to be performed based on the update time of the data. For example, among candidate data for post-process compression, CPU 107 may prioritize data with the oldest update time and perform post-process compression on the data. Candidate data is valid data for which post-process compression has not yet been performed. CPU 107 may select the oldest data first, or may select data for which post-process compression is to be performed from data for which the elapsed time since update exceeds a threshold.

[0146] In step 1003, the LZMA compression / decompression circuit 104 decompresses the inline-compressed data stored in the cache memory 106 in response to an instruction from the CPU 107, and stores the decompressed data in the cache memory 106. By using the LZMA compression / decompression circuit 104, data can be decompressed efficiently in a short time.

[0147] In step 1004, CPU 107 executes dictionary compression 171, which has stronger string search capabilities than inline compression. This allows for a higher compression rate than inline compression. For example, CPU 107 executes dictionary compression using a hash table with a large number of hash bits or a large upper limit on the number of entries. Alternatively, CPU 107 may execute dictionary compression using a hash table with a small number of characters constituting a string.

[0148] The hash table may have both a larger number of hash bits and an upper limit on the number of entries than those in inline compression. The hash table may have a larger number of hash bits and / or an upper limit on the number of entries than those in inline compression, and the number of characters constituting a string may be smaller than those in inline compression. The hash table may be stored in memory within the storage controller 103.

[0149] In step 1005, the CPU 107 performs range encoding 172, which is compatible with the range encoding 203 of the LZMA compression / decompression circuit 104, to encode the data into multiple subcodes. This enables high-speed processing using the LZMA compression / decompression circuit 104 when reading compressed data.

[0150] In step 1006, the CPU 107 stores the output code (post-process compressed data (post-compressed data in FIG. 11)) in the cache memory 106. In step 1007, the CPU 107 writes the post-process compressed data to an address area in the SSD 105 that is different from the address area of ​​the original inline compressed data. In step 1008, the CPU 107 invalidates the selected inline compressed data and validates the post-process compressed data. Specifically, the CPU 107 updates management information that manages the address of the SSD 105 and whether the stored data is valid or invalid.

[0151] In addition, in step S1006, the post-process compressed data may be stored in a cache memory, and then the data may be transferred and stored in a storage area on the cloud in parallel with the operation of writing the post-process compressed data in the cache memory to SSD 105, thereby taking a backup.

[0152] In addition, by configuring the system to limit backup to the cloud when writing inline compressed data to SSD 105, it is possible to reduce cloud capacity usage and achieve enhanced data preservation performance in a hybrid environment that combines an on-premise system and a cloud system. Note that data transfer to the cloud may be at a time other than the above, and the entire system may be configured as a hybrid cloud system so that post-process compressed data is transferred periodically.

[0153] Furthermore, with regard to the backup function that utilizes the storage capacity on the cloud, it is possible to adopt a mode in which the process from S1006 onwards (storing post-process compressed data) is applied only to data storage on the cloud. That is, in S1006, the post-process compressed data is stored in cache memory, and then the post-process compressed data is transferred to a storage area on the cloud.

[0154] In this case, the selection of data to which post-process compression is applied can be processed based on, for example, the most recent frequency of use of the inline-compressed data. For example, the above-mentioned frequency of use is monitored, and data whose frequency of use is lower than a predetermined frequency (i.e., data with low frequency of use) is identified as data to which post-process compression is applied, and such data is transferred to the cloud after post-process compression, and the corresponding inline-compressed data is invalidated according to the process of S1008.

[0155] With this hybrid system, frequently used data can be stored in the on-premises environment to ensure responsiveness, while less frequently used data can be stored in the cloud environment with a high compression ratio. In other words, it is possible to achieve both responsiveness for reading and writing data and effective use of storage space. [Example]

[0156] Another embodiment of post-process compression will be described. Differences from the first embodiment will be mainly described below. The description of the first embodiment can be applied to a configuration similar to that of the first embodiment. Fig. 12 is a logical configuration diagram for explaining the other embodiment of post-process compression, and Fig. 13 is a flowchart for explaining the other embodiment of post-process compression.

[0157] 12 , in the post-process compression of this embodiment, data dictionary-compressed by the CPU 107 is compressed by the range encoding 203 of the LZMA compression / decompression circuit 1104. This reduces the load on the CPU 107 and speeds up the compression process. The LZMA compression / decompression circuit 1104 includes a selector 209 in addition to the configuration of the LZMA compression / decompression circuit 104 of the first embodiment. The selector 209 selects the dictionary compression 202 in inline compression, and selects the bit string 215 resulting from the dictionary compression by the CPU 107 in post-process compression, and outputs it to the range encoding 203.

[0158] 13, steps 1201 to 1204 are the same as steps 1001 to 1004 in the flowchart of Fig. 11. In step 1205, CPU 107 compresses the dictionary-compressed data using range encoding 203 of LZMA compression / decompression circuit 1104. Steps 1206 to 1208 are the same as steps 1006 to 1008 in the flowchart of Fig. 11.

[0159] In addition, in step S1206, the post-process compressed data may be stored in a cache memory, and then the data may be transferred and stored in a storage area on the cloud in parallel with the operation of writing the post-process compressed data in the cache memory to SSD 105, thereby taking a backup.

[0160] In addition, by configuring the system to limit backup to the cloud when writing inline compressed data to SSD 105, it is possible to reduce cloud capacity usage and achieve enhanced data preservation performance in a hybrid environment that combines an on-premise system and a cloud system. Note that data transfer to the cloud may be at a time other than the above, and the entire system may be configured as a hybrid cloud system so that post-process compressed data is transferred periodically.

[0161] Furthermore, with regard to the backup function that utilizes the storage capacity of the cloud, it is possible to adopt a mode in which the process from S1206 onwards (storing post-process compressed data) is applied only to data storage on the cloud. That is, in S1206, the post-process compressed data is stored in cache memory, and then the post-process compressed data is transferred to a storage area on the cloud.

[0162] At this time, it is also possible to select data to which post-process compression is applied based on, for example, the most recent frequency of use of inline-compressed data. For example, the above-mentioned frequency of use is monitored, and data whose frequency of use is lower than a predetermined frequency (i.e., data with low frequency of use) is identified as data to which post-process compression is applied, and such data is transferred to the cloud after post-process compression, and the corresponding inline-compressed data is invalidated according to the process of S1208.

[0163] With this hybrid system, frequently used data can be stored in the on-premises environment to ensure responsiveness, while less frequently used data can be stored in the cloud environment with a high compression ratio. In other words, it is possible to achieve both responsiveness for reading and writing data and effective use of storage space.

[0164] Furthermore, the storage systems of the first and second embodiments and their modifications can reduce the amount of data, thereby reducing the storage capacity used and thereby saving resources by reducing the number of storage drives, and reducing the power consumption of the storage drives.

[0165] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0166] Furthermore, the above-mentioned components, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned components, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in memory, a storage device such as a hard disk or SSD, or a storage medium such as an IC card or SD card.

[0167] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0168] 101...storage system, 103...storage controller, 104, 1104...LZMA compression / decompression circuit, 107 CPU, 171 dictionary compression, 172 range encoding, 301, 601A-D...encoders, 311, 8A, 8B0, 8B1, 8C00, 8C11, 8D000, 8D111...decoders, 304, 314, 604, 804...probability tables, 302, 602...input bit strings, 303, 606...output codes, 312, 802...input codes, 313, 806...output bit strings

Claims

1. 1. A storage system, comprising: a controller including a processor and a data compression and decompression circuit; the controller performs inline compression of plaintext data from a host and post-process compression of the inline compressed data stored on one or more storage drives; The inline compression The data compression and decompression circuit performs a compression process including a first dictionary compression on the plaintext data from the host to generate inline compressed data; storing the inline compressed data on the one or more storage drives; The post-process compression decompressing the inline compressed data read from the one or more storage drives by the data compression and decompression circuit to generate the plaintext data; A storage system comprising: performing a compression process on the plain text data, including a second dictionary compression by the processor, the second dictionary compression having better string search capabilities than the first dictionary compression, to generate post-processed compressed data and store the data on the one or more storage drives.

2. 2. The storage system according to claim 1, the compression processing by the data compression / decompression circuit includes, after performing the first dictionary compression, encoding using a first compression / decompression algorithm different from that used for the first dictionary compression; The post-process compression is performed by compressing the plaintext data using the second dictionary compression by the processor, and then performing encoding using the first compression / decompression algorithm to generate the post-process compressed data.

3. 3. The storage system according to claim 2, A storage system in which the post-process compression is performed by the processor using the first compression / decompression algorithm.

4. 3. The storage system according to claim 2, In the post-process compression, data compressed by the second dictionary compression is input to the data compression / decompression circuit, and encoding is performed using the first compression / decompression algorithm.

5. 2. The storage system according to claim 1, The storage system, wherein the controller starts the post-process compression when the utilization rate of the processor is less than a preset threshold.

6. 2. The storage system according to claim 1, The post-process compression selects the oldest data among the candidate data stored on the one or more storage drives.

7. 2. The storage system according to claim 1, A storage system in which at least one of the number of hash bits and the upper limit of the number of entries in the hash table for the second dictionary compression is larger than that in the hash table for the first dictionary compression, and / or the number of characters in a string in the hash table for the second dictionary compression is smaller than that in the hash table for the first dictionary compression.

8. 2. The storage system according to claim 1, The storage system, wherein the controller performs the post-process compaction during garbage collection of the one or more storage drives.

9. 3. The storage system according to claim 2, The storage system wherein the controller reads the data compressed by the post-process compression from the one or more storage drives, decompresses the data into the plain data using the data compression / decompression circuit, and transmits the plain data to the host.

10. A data compression method in a storage system, comprising: The storage system includes: a processor and a data compression and decompression circuit, The data compression method includes: inline compression of plaintext data from the host and post-process compression of the inline compressed data stored on one or more storage drives; The inline compression The data compression and decompression circuit performs a compression process including a first dictionary compression on the plaintext data from the host to generate inline compressed data; storing the inline compressed data on the one or more storage drives; The post-process compression decompressing the inline compressed data read from the one or more storage drives by the data compression and decompression circuit to generate the plaintext data; A data compression method comprising: performing a compression process on the plain text data, the compression process including a second dictionary compression by the processor, the second dictionary compression having better string search capability than the first dictionary compression, to generate post-processed compressed data, and storing the post-processed compressed data on the one or more storage drives.

Images