Storage system and data compression method

Abstract

The invention provides a storage system and a data compression method for improving a data compression rate. The controller performs inline compression of the plaintext data from the host and post-processing compression of the inline compressed data stored in the one or more storage drives. In the inline compression, compression processing including first dictionary compression is performed on plaintext data from a host by a data compression / decompression circuit to generate inline compressed data, and the inline compressed data is stored in one or more storage drivers. In post-processing compression, the data compression / decompression circuit decompresses the inline compressed data read from the one or more memory drivers to generate plaintext data. In the post-processing compression, compression processing including second dictionary compression performed by the processor is performed on the plaintext data to generate post-processing compressed data, and the post-processing compressed data is stored in the one or more memory drives, and the character string search capability of the second dictionary compression is better than that of the first dictionary compression.

Description

Technical Field

[0001] The present invention relates to a data compression technology in a storage system. Background Art

[0002] Storage systems, which are information devices used to store and manage large amounts of data, can reduce unit costs by storing more data. Therefore, some storage systems are equipped with a function to compress written data before storing it in a disk drive.

[0003] For example, in recent years, SSDs (Solid State Drives) equipped with NAND flash memory, a non-volatile semiconductor memory, have been used as storage media in addition to or instead of HDDs (Hard Disk Drives). Because SSDs lack the physical head seek mechanism used in data access, unlike HDDs, they experience less head-positioning latency and offer excellent responsiveness during random data reads.

[0004] Therefore, in applications such as databases that require high-speed random access, the storage system is shifting from HDDs to SSDs as storage media. However, while the cost per bit of SSDs has been decreasing year by year due to the increasing integration of flash memory cells, it is still higher than the cost per bit of HDDs.

[0005] Therefore, most storage systems using SSDs as storage media have the ability to implement reversible compression technology to reduce the size of data stored on the SSD. This virtually increases the system's storage capacity, reduces the cost per unit of capacity, and brings it closer to storage systems using HDDs as storage media.

[0006] Patent document 1 serves as background technology for the present disclosure. Patent document 1 discloses a storage system using the LZMA algorithm. In the LZMA compression process, dictionary compression is first performed on the plaintext data before compression. Then, interval encoding is performed on the dictionary compression result. Thus, LZMA compressed data is generated. In the LZMA decompression process, interval decoding is first performed on the compressed data. Then, plaintext expansion is performed on the decoding result. Thus, the original plaintext data is generated. Patent document 1 makes the LZMA interval code consist of multiple intervals, and by making the interval encoder / decoder perform parallel processing, inline compression (inline compression) and decompression speed up.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-095015 Summary of the Invention

[0010] Problems to be solved by the invention

[0011] Inline compression, which compresses data received from the host and stores it in the storage drive, requires high compression ratios and meets the storage system's I / O performance requirements. Data compression using hardware circuits (accelerators) can achieve high-speed data compression, thus meeting higher I / O performance requirements. However, storage system design also places limitations on the scale of the accelerator's FPGA (Field Programmable Gate Array) logic.

[0012] Therefore, there is a need for a technology that improves the compression ratio of stored data while satisfying the IO performance requirements of the storage system and the limitations on the circuit scale for data compression.

[0013] Means used to solve problems

[0014] A storage system according to a technical solution of the present invention includes a controller, which includes a processor and a data compression and decompression circuit. The controller performs inline compression of plaintext data from a host and post-processing compression of the inline compressed data stored in one or more storage drives. The inline compression includes: the data compression and decompression circuit performs compression processing including a first dictionary compression on the plaintext data from the host to generate inline compressed data; the inline compressed data is stored in the one or more storage drives. The post-processing compression includes: the data compression and decompression circuit decompresses the inline compressed data read from the one or more storage drives to generate the plaintext data; the processor performs compression processing on the plaintext data to generate post-processing compressed data, and the data is stored in the one or more storage drives. The string search capability of the second dictionary compression is better than that of the first dictionary compression.

[0015] Effects of the Invention

[0016] According to a technical solution of the present invention, data stored in a storage system can be compressed more effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This section shows an example of a storage system configuration.

[0018] Figure 2A This section summarizes the LZMA algorithm.

[0019] Figure 2B A specific example of dictionary compression processing is shown.

[0020] Figure 2CThis shows an example of the structure of a hash table.

[0021] Figure 3A Functional block diagram showing interval encoding.

[0022] Figure 3B A functional block diagram showing interval decoding.

[0023] Figure 4A An example is shown to explain the principle of the interval code.

[0024] Figure 4B Another example for explaining the principle of the interval code is shown.

[0025] Figure 5A A flowchart showing the section encoding process.

[0026] Figure 5B This is a flowchart showing the section decoding process.

[0027] Figure 6 This is a functional block diagram showing a method for speeding up section encoding processing.

[0028] Figure 7 A flowchart showing a method for speeding up section encoding processing.

[0029] Figure 8 This is a functional block diagram showing a method for speeding up section decoding processing.

[0030] Figure 9 A flowchart showing a method for speeding up section decoding processing.

[0031] Figure 10 A diagram illustrating the logical structure of post-processing compression.

[0032] Figure 11 Flowchart illustrating post-processing compression.

[0033] Figure 12 This is a logical configuration diagram for explaining another example of post-processing compression.

[0034] Figure 13 A flowchart illustrating another example of post-processing compression.

[0035] Description of labels

[0036] 101…storage system; 103…storage controller; 104, 1104…LZMA compression and decompression circuit; 107…CPU; 171…dictionary compression; 172…interval coding; 301, 601A–D…encoder; 311, 8A, 8B0, 8B1, 8C00, 8C11, 8D000, 8D111…decoder; 304, 314, 604, 804…probability table; 302, 602…input bit string; 303, 606…output code; 312, 802…input code; 313, 806…output bit string. DETAILED DESCRIPTION

[0037] For the sake of convenience, the following description is divided into multiple parts or embodiments as necessary. However, unless otherwise specified, they are not mutually exclusive, but rather one part is a modification, detail, supplementary explanation, etc. of part or all of the other. In addition, when the number of elements (including number, value, amount, range, etc.) is mentioned below, except for cases where it is specifically specified or where it is clearly limited to a specific number in principle, it is not limited to the specific number and may be greater than or less than the specific number.

[0038] In one embodiment of this specification, an accelerator (logic circuit) compresses data written to a storage system during inline processing and writes it to a storage drive. Furthermore, the inline compressed data written to the storage drive is restored to plaintext during post-processing, compressed again using a compression method that enhances dictionary compression compared to the accelerator, and written back to the storage drive. This improves the data compression rate of the storage system (further reducing the amount of stored data) without increasing the circuit scale of the accelerator.

[0039] LZMA compression improves compression ratios by enhancing the search capabilities of dictionary compression. However, using hardware circuits (accelerators) for LZMA compression increases the logic size of the FPGA (Field Programmable Gate Array). Furthermore, the dictionary compression performed by software in inline compression can degrade I / O performance.

[0040] Considering the storage system's I / O performance and the accelerator's FPGA logic scale, it's difficult to enhance dictionary compression to improve inline compression rates. Therefore, a technology is needed that can improve the compression rate of stored data while meeting the storage system's I / O performance and the circuit scale constraints for data compression.

[0041] [Example 1]

[0042] (1) System composition

[0043] Hereinafter, a storage system having a data compression function will be described as one embodiment of this specification. The storage system reduces the amount of stored data by lossless compression.

[0044] Figure 1 The following shows an example of the configuration of a storage system according to one embodiment of this specification. A storage system 101 includes a host interface 102, a storage controller 103, multiple solid state drives (SSDs) 105, and a cache memory 106 using volatile memory such as DRAM (Dynamic Random Access Memory).

[0045] Storage controller 103 includes CPU 107, a processor connected to host I / F 102, SSD 105, and buffer memory 106 to control them, and memory 109. CPU 107 may include one or more cores and operates as a predetermined functional unit by executing programs (software) stored in memory 109.

[0046] The memory 109 stores system software including an operating system required for programs to run on the storage controller 103, programs running on the processor 107, metadata used by the programs, and data that needs to be temporarily stored. Alternatively, only one of the memory 109 and the buffer memory 106 may be installed, and the data of the other may also be stored.

[0047] The CPU 107 interprets the contents of read / write commands from a host (not shown), transmits and receives data to and from the host, compresses and decompresses data using the LZMA compression / decompression circuit 104, which is a data compression / decompression circuit, and transfers data to and from the SSD 105 or buffer memory 106. The CPU 107 also performs post-processing compression on data stored in the SSD 105.

[0048] The host I / F 102 is an interface mechanism for connecting to an external host and responds to read / write commands to send or receive data from the host. The mechanism of the host I / F 102 and the protocol for sending and receiving commands and data conform to, for example, standard interface specifications.

[0049] The storage controller 103 includes an LZMA compression / decompression circuit 104 and a transmission circuit 108. The transmission circuit 108 receives and sends data compressed or decompressed by the LZMA compression / decompression circuit 104. The transmission circuit 108 transmits data between components of the storage system 101, for example, between the LZMA compression / decompression circuit 104 and the buffer memory 106, and between the CPU 107 and the buffer memory 106. The transmission circuit 108 also transmits data between the host I / F 102 and the buffer memory 106, and between the SSD 105 and the buffer memory 106.

[0050] The LZMA compression / decompression circuit 104 reversibly compresses received write data in response to a write command to reduce the amount of data stored in the SSD 105, which serves as a storage drive, to generate compressed data. Furthermore, in response to a read command, the circuit decompresses compressed data read from the SSD 105 to generate plaintext data in order to transmit the original plaintext data to the host.

[0051] The storage controller 103 performs inline compression and post-processing compression. Inline compression compresses the write data from the host and stores it in the SSD 105. Post-processing compression reads the compressed data from the SSD 105, recompresses the decompressed data, and sends it back to the SSD 105.

[0052] In inline compression, write data from the host is first temporarily stored in buffer memory 106. At this point, the storage controller 103 notifies the host of the completion of the write. The data is then converted into compressed data by the LZMA compression / decompression circuit 104, which is also temporarily stored in buffer memory 106. The compressed data is then written to SSD 105. As described below, the LZMA compression / decompression circuit 104 performs dictionary compression and range encoding.

[0053] Post-processing compression reads compressed data from SSD 105 and stores it in buffer memory 106. Next, post-processing compression uses LZMA compression and decompression circuit 104 to decompress the compressed data and restore it to plaintext. CPU 107 then compresses the data again using dictionary compression and range encoding, which further enhance the string search capabilities of LZMA compression and decompression circuit 104. The compressed data is temporarily stored in buffer memory 106 before being written to SSD 105.

[0054] Data to be read from the host is read from the SSD 105 in a compressed state and first temporarily stored in the buffer memory 106. It is then converted into plaintext data by the LZMA compression / decompression circuit 104 and the plaintext data is also temporarily stored in the buffer memory 106. The plaintext data is then sent to the host.

[0055] The LZMA compression and decompression circuit 104 is implemented as hardware (logic circuitry) designed based on the data compression and decompression method of one embodiment of this specification, as opposed to a processor operating via software. For example, it can be an accelerator using an FPGA. By enabling the LZMA compression and decompression circuit 104 to possess high-speed data decompression performance, the storage system 101 can utilize the high-speed random read performance that is a hallmark of SSDs not only for uncompressed data but also for compressed data.

[0056] A storage drive different from an SSD, such as an HDD (Hard Disk Drive), may also be used. The storage drive is not installed in the housing of the storage system 101 but may be connected to the storage controller 103 or the buffer memory 106 via a network, or the storage area of ​​the storage drive may exist in the cloud.

[0057] (2) LZMA algorithm

[0058] As a prerequisite for explaining the data decompression method of one embodiment of this specification, Figures 2A to 5B The LZMA algorithm is described.

[0059] (2-1) Overview of the LZMA Algorithm

[0060] Figure 2A 1 shows an overview of the LZMA algorithm. In the LZMA compression process, dictionary compression 202 is first performed on uncompressed plaintext data 201. Then, interval encoding 203 is performed on the dictionary compression result. This generates LZMA compressed data 204.

[0061] On the other hand, in the LZMA decompression process, the compressed data 204 is first subjected to the interval decoding process 205. Then, the decoding result is subjected to the plaintext expansion process 206. In this way, the original plaintext data 201 is generated.

[0062] (2-2) Dictionary Compression Processing

[0063] Figure 2B This figure shows a specific example of dictionary compression 202, which constitutes the LZMA algorithm. In the string stream of plaintext data 201, the same string is checked sequentially to see if it appears again. If a string contains L characters that are identical starting from the first character and continuing from the Jth character before, the string is converted into a duplicate marker [L, J].

[0064] For example, a string 211 of four characters, "b, c, d, e," has four characters that are consecutive and identical starting with the first character "b" and starting from the first six characters. In this case, string 211 is converted to the duplicate mark [4, 6]. Similarly, a string 212 of four characters, "a, b, a, b," has four characters that are consecutive and identical starting with the first character "a" and starting from the first two characters (including any overlapping portions). In this case, string 212 is converted to the duplicate mark [4, 2].

[0065] Similarly, the six-character string 213 of "c, d, e, f, e, b" has six consecutive characters starting with the first character "c" and continuing for 15 characters. In this case, string 213 is converted to the duplicate markers [6, 15]. The amount of data in these duplicate markers is smaller than that of the original string, so this conversion can reduce the data volume.

[0066] The string stream referenced during a match search (hereinafter referred to as the dictionary) is set to the range from one character before to a specified number of characters before. Because the dictionary range slides backward with each search, this compression technique is also known as sliding character representation compression. Furthermore, if there are multiple matching strings within the dictionary range, the longest consecutive matching string is converted into a duplicate marker. This further reduces the amount of data.

[0067] To generate data to be input to the subsequent interval code 203, characters not converted into copy marks (hereinafter referred to as text characters) and copy marks must be encoded in a predetermined bit pattern and concatenated to form a bit stream.

[0068] Figure 2B A bit stream representing the result of encoding according to the rules of the LZMA specification is input to interval code 203. For example, bit pattern 221 represents the copy flag [4, 6] with a length of 12 bits. Bit pattern 222 represents the copy flag [4, 2] with a length of 11 bits. Bit pattern 223 represents the copy flag [6, 15] with a length of 13 bits. Thus, the length of the bit pattern corresponding to the copy flag is not fixed. On the other hand, a text character is represented by a 9-bit bit pattern with a 0 added to the beginning of the 8-bit value of the character.

[0069] The section decoding process 205 outputs such a bit stream in the LZMA decompression process. In the plaintext expansion process 206, if such a bit stream is input, it is interpreted as a copy mark or a character string, and the character string stream of the plaintext data 201 is restored.

[0070] Figure 2CThe figure shows an example of the structure of the hash table 230 used by the LZMA compression / decompression circuit 104 to search for a matching string through the dictionary compression 202. The hash table 230 shows the three-character strings that can obtain each hash value in the input data and the positions of the strings. Figure 2C In the example shown, each hash value is represented by 3 bits, and the character string is 3 consecutive characters. In addition, the upper limit of the number of entries of each hash value is 3.

[0071] The following describes the processing sequence of dictionary compression 202. Figure 2B As shown, the LZMA compression / decompression circuit 104 has the following structure: “a, b, c, d, e, f, e, b, c, d, e, a, b, a, b, a, b, c, d, e, f, e, b, ...” The LZMA compression / decompression circuit 104 performs the following processing in sequence from the beginning of the input data.

[0072] [1] First, the LZMA compression / decompression circuit 104 calculates a hash value X (3 bits) of three characters starting from the current position, and registers the current position and the three characters in the hash table 230. The characteristics of the hash function allow the same hash value to be calculated from character strings that are not completely identical.

[0073] [2] Next, if the number of entries with hash value X exceeds three, the LZMA compression / decompression circuit 104 discards the oldest entry (the entry with the smallest value indicating the position). Frequently occurring character strings are frequently registered in entries, so the probability of hitting an entry with the same character string is high in step [3] below. On the other hand, infrequently occurring character strings have a low priority from the perspective of compression efficiency achieved by converting them to duplicate symbols. Therefore, by preferentially excluding such character strings from entries, efficient use of the hash table is achieved.

[0074] [3] Next, the LZMA compression / decompression circuit 104 refers to other entries related to the hash value X and checks whether there is an entry that stores a string that matches the newly registered current string. In addition, if the strings stored in one or more entries for the hash value X are the same, the string closest to the current string is selected. Regarding this selection operation, if it is assumed that, for example, Figure 2C If abc is again given as the current string, the following can be understood. The new current string abc is registered in entry 3, and entry 2 containing a string that matches the current string is identified. On the other hand, if a hash value X is calculated but no string matching the current string exists in the corresponding entry, the process proceeds to step [7].

[0075] [4] Next, since there's a possibility that more than four characters may match, the LZMA compression / decompression circuit 104 directly reads the input data and checks for the longest matching length L. Specifically, the current string is checked for matching characters in the entry associated with the corresponding hash value X, and the matching range is also checked for subsequent characters following that string. As a result, a matching string (three or more characters) starting with the string stored in the entry is identified at two locations, and the length of this identified string is calculated as the longest matching length L.

[0076] [5] Next, the LZMA compression / decompression circuit 104 determines the distance D from the current position (the position of the current string) to the beginning of the found identical string (equal to the difference between the positions stored in the two entries being compared).

[0077] [6] Next, the LZMA compression / decompression circuit 104 converts the character string of length L from the current position into a copy symbol [L, D] and proceeds to step [8].

[0078] [7] If there is no character string that matches the current character string in step [3], the LZMA compression / decompression circuit 104 converts one character at the current position into a character symbol.

[0079] [8] After step [6], if the end of the data is reached, the LZMA compression and decompression circuit 104 ends the process. Otherwise, the LZMA compression and decompression circuit 104 returns to step [3].

[0080] The dictionary notation of the above plain text data becomes as follows: “a, b, c, d, e, f, e, [4, 6], a, b, [4, 2], [6, 15], …”.

[0081] As described above, a string consisting of three matching characters can be efficiently found by using the hash table 230. Then, by checking for matches of four or more characters, matching strings consisting of even more characters can be found.

[0082] Generally, the larger the number of bits of the hash value (the number of rows in the hash table 230) and the larger the upper limit of the number of entries (the number of columns in the hash table 230), the more the strings registered in the hash table 230 increase by a power of 2, and the probability of finding a consistent string increases. That is, the compression rate of the dictionary compression increases. However, in order to improve the compression rate, a large-capacity table memory is required, and the circuit scale of the LZMA compression and decompression circuit 104 becomes larger. By reducing the number of characters in the string, the probability of finding a consistent string can also be increased. However, the search time for a consistent string in plaintext data increases. In addition, by increasing the number of entries allowed for the hash value, the probability of finding a consistent string can also be increased. That is, even if the frequency of occurrence is low, the chance of finding a consistent string is increased by lengthening the period for maintaining the string and position in the entry. However, the adoption of this method also increases the search time for a consistent string.

[0083] (2-3) Section Encoding / Decoding Processing

[0084] Figure 3A Functional block diagram showing interval coding, Figure 3B This is a functional block diagram showing section decoding. Figure 3A The interval encoding function 300 will be described. The encoder 301 is a computation block that receives input from an input bit string 302 in 1-bit units and generates an output code 303. Figure 3A In the example, the input bit string is "1, 1, 0, 1", which is equivalent to Figure 2B Regarding the method of generating the output code 303, use Figure 4A and 4B This will be explained in the next section. Output code 303 is equivalent to the compressed data of the LZMA algorithm ( Figure 2A LZMA compressed data 204).

[0085] The encoder 301 also uses a probability value referenced from a probability table 304 as input. This probability value P(x) represents the probability that the next input bit from the input bit string 302 is "0" when the currently input bit history 305 is x. In addition, in order to simplify the description, a general probability expression using a decimal point is adopted in this figure, but in practice, a sufficiently large range of integers is given, and the range is divided into integer widths (divisions) corresponding to the probability. In addition, in the case where the range of integers is too small to be suitable for further division, the number of bits is increased at this stage so that it can be divided again, thereby avoiding problems such as floating point decimals. In addition, the probability value P(x) can also represent the probability that the next input bit from the input bit string 302 is "1" when the currently input bit history 305 is x.

[0086] Encoder 301 performs learning-based adaptation each time a probability value P(x) is used. For example, if the actual next input bit is "0," P(x) is increased, and if the actual next input bit is "1," P(x) is decreased. At the start of encoding, all unused P(x) values ​​are 0.5 (the probabilities of "0" and "1" are equal).

[0087] For example, in Figure 3A In the example, if the first bit is given, then at that time point x = "-". Moreover, the probability that the first bit is "0" is 0.4, and the probability that it is "1" is 0.6. In addition, when the encoder is completely initialized and a bit string is given, P(-) = 0.5 (initial value), but Figure 3A The result P "-" of learning using the bit history is different from the initial value.

[0088] In the case of x=0, when the first bit is "0", the probability of the second bit being "0" is 0.35, and the probability of being "1" is 0.65. In this case, if the second bit is "0", the probability of P(x=0) is increased, and if the second bit is "1", the probability of P(x=0) is reduced. These probabilities are updated through learning of the entire bit history (bit stream) as the object. In addition, using Figure 4A The mechanism of compression performed by applying the probability table 304 obtained in this way to the bit history 305 will be described.

[0089] Next, refer to Figure 3B The interval decoding function 310 is described. The decoder 311 is a computation block that receives an input code 312 (i.e., an output code 303) and generates an output bit string 313 in units of 1 bit. Figure 4A and 4B The output bit stream 313 corresponds to the input bit stream to the plaintext expansion process (plaintext expansion process 206 in FIG2 ) of the LZMA algorithm, that is, the output bit stream from the dictionary compression 202 in FIG2 .

[0090] The decoder 311 also uses as input a probability value referenced from the probability table 314. This probability value P(x) represents the probability that the next output bit will be "0" when the bit history 315 currently output is x.

[0091] Similar to encoder 301, decoder 311 performs learning-based adaptation each time a probability value P(x) is used. For example, if the actual next output bit is "0," P(x) is increased, and if the actual next output bit is "1," P(x) is decreased. At the start of decoding, all unused P(x) values ​​are 0.5 (the probabilities of "0" and "1" are equal).

[0092] When the output code 303 of the interval encoding function 300 is the same as the input code 312 of the interval decoding function 310, the changes in probability values ​​P(x) due to learning are the same in the probability tables 304 and 314. Therefore, the changes during encoding are reproduced during decoding.

[0093] (2-4) Principles of interval encoding / decoding

[0094] Figure 4A and 4B An example is shown to explain the principle of interval encoding / decoding. Figure 3A The encoding process performed by encoder 301 repeatedly performs the following processing: the value axis [0, 1) is divided into segments corresponding to the probability that each bit value of the input bit string is "0", and one of the segments is retained as the segment target for the next bit. The encoding process outputs the coordinate values ​​contained in the segment that is finally retained as a code. In addition, if the bit is "0", the segment on the left of the segment is retained, and if the bit is "1", the segment on the right of the segment is retained. In addition, the probability of each bit value being "0" is obtained from probability table 304 using the current input bit history as an index.

[0095] According to the LZMA algorithm, the bit history used to reference probability values ​​is cleared under specified conditions. For example, of the 9 bits representing a character, encoder 301 encodes the first bit of the 8 bits, excluding the first bit, using a NULL bit history. For the final 8 bits, encoder 301 encodes bits 1 to 7 as a bit history and then clears the bit history.

[0096] Figure 3B The decoding process performed by the decoder 311 is as follows Figure 4A 、 4B As shown, the value axis [0, 1) is segmented into lengths corresponding to the probability that each bit value in the output bit string is "0." Decoder 311 then repeatedly performs the following process: checking which segment the input code (coordinate value) is contained in, determining each bit in the output bit string, and retaining the included segment as the target for segmentation for the next bit. Ultimately, all values ​​in the output bit string are determined.

[0097] Furthermore, when segmentation is performed, if the left segment contains a code, decoder 311 determines that the bit is "0," and if the right segment contains a code, the bit is "1." Furthermore, the probability of each bit value being "0" is obtained from probability table 314 using the current output bit history as an index.

[0098] Decoder 311 clears the bit history used to reference the probability value under the same conditions as encoding. For example, of the 9 bits representing a character, decoder 311 decodes the first bit of the 8 bits, excluding the first bit, using a NULL bit history. For the final 8th bit, decoder 311 decodes bits 1 to 7 as a bit history and then clears the bit history.

[0099] Figure 4A and 4B Both represent examples of transitions in the numerical axis division in the encoding / decoding process of the bit string "1101". However, the probability values ​​of the bit value being "0" are different. Figure 4A In the example, the probability of a bit value being "0" is always 0.5. Figure 4B In the example, the probability values ​​of the bit value being "0" are 0.25, 0.25, 0.75, and 0.25 in bit order. Figure 4A This shows a case where the probability values ​​referred to from the probability tables 304 and 314 are in an initial state.

[0100] Therefore, if the probability table 304 has been learned, the segments are divided according to the probabilities stored in the learned probability table 304. In addition, by inputting the bit string "1101", Figure 4A In the 1st, 2nd, and 4th columns, the "1" blocks are made larger than the "0" blocks, and in the 3rd column, the "0" blocks are made larger than the "1" blocks. Furthermore, the size of the "0" blocks in each row (the value that becomes the boundary with "1" when the left end of each row is considered to be 0) is registered in probability table 304. (Also, when encoding the bit string "1101", P(x) in the rows where x = "-", "1", "10", and "110" is updated.) Figure 4B This shows how the probability values ​​referenced from the probability tables 304 and 314 change through learning.

[0101] During interval encoding / decoding, if the bit string "1101" is frequently processed, probability tables 304 and 314 are trained to predict that "1" is likely to appear initially, then "1" is likely to appear if the history is "11," then "0" is likely to appear if the history is "110," and then "1" is likely to appear if the history is "110." This probability learning continues, and if the predicted bit occurrence pattern is correct, the output code of the interval encoding becomes shorter.

[0102] Figure 4A The encoding process is as follows:

[0103] ■Step 1: According to input “1”, keep the right 1 / 2 segment [1 / 2~2 / 2).

[0104] ■Step 2: According to input “1”, retain the right 1 / 2 section [3 / 4~4 / 4).

[0105] ■Step 3: Enter “0” to retain the left 1 / 2 section [6 / 8~7 / 8).

[0106] ■Step 4: Enter “1” to retain the right 1 / 2 section [13 / 16~14 / 16).

[0107] ■Use 13 / 16 (binary number 1101) contained in the last segment as output code 401.

[0108] on the other hand, Figure 4B The encoding process is as follows:

[0109] ■Step 1: Enter “1” and keep the right 3 / 4 section [1 / 4~4 / 4).

[0110] ■Step 2: Enter “1” to retain the right 3 / 4 section [7 / 16~16 / 16).

[0111] ■Step 3: Enter “0” to retain the left 3 / 4 segment [28 / 64~55 / 64).

[0112] ■Step 4: Enter “1” to retain the right 3 / 4 segment [139 / 256~220 / 256).

[0113] ■Use 3 / 4 (binary number is 11) contained in the last segment as the output code 411.

[0114] The more the bit string is input as predicted based on probability (i.e., the more similar it is to the bit history processed in the past), the larger the size of the segment retained during segmentation. Therefore, the number of bits required to represent the coordinate values ​​of the output code contained in the segment finally retained can be reduced. Figure 4A In the example, since the probability table is the initial value, 4 bits are required; in contrast, Figure 4B In the example, the result of applying the learned probability table can be 2 bits. In this way, interval coding improves the compression rate by learning the bit occurrence probability corresponding to the bit history.

[0115] In addition, the output codes 401 and 411 obtained as a result of the above encoding are given as input codes in the decoding process. For example, when the output code 401 is given as the input code, first obtain P(x) when x = "-" from the probability table 314. Next, determine in which division of "0" and "1" the probability of P(x = "-") is included. In addition, if it is Figure 4AIn the case of , P(x) is 0.5 for any x, so P(x="-") is calculated as 1 / 2, and it is judged that 13 / 16 exists in the range of 1 / 2 to 2 / 2 after the 0 / 2 to 2 / 2 area is equally divided. As a result, the first bit is determined to be "1".

[0116] Next, since the first bit has been determined, probability table 314 is referenced to obtain P(x="1") = 0.5, and it is determined in which of the "0" and "1" bins the 13 / 16 bits of output code 401 are contained. In this case, since 13 / 16 is contained within the 2 / 4 to 4 / 4 bins, which falls within the 3 / 4 to 4 / 4 range, the second bit is determined to be "1." Next, using the bit history (x = 11), the probability table is referenced to re-determine the bin containing output code 401 and determine the corresponding bit value. By repeating this process, the input bit string can be decoded from the output code.

[0117] The probability for each bit history (x) in probability table 314 is updated each time a bit is decoded, and the updated content follows the same process as when encoder 301 generates output code 303. Specifically, when input code 312 is decoded from the beginning (i.e., determined from the beginning bit of bit history 315), at the initial stage, similar to the initial stage of encoder 301 generating output code 303, segmentation is performed based on probability information close to the initial value, and bit history 315 containing input code 312 is determined. If output code 303 is the same as input code 312, the bit history determined during segmentation will also be the same.

[0118] Next, as decoding progresses, probability table 314 is updated, and probability information corresponding to the characteristics of bit history 315 appears. The segment division corresponding to the probability information learned based on the characteristics of bit history 315 is the same as the segment division during encoding. Therefore, if input code 312 and output code 303 are the same, the bit string determined by the calculated segment division is uniquely determined, and input bit string 302 can be decoded into output bit string 313.

[0119] (2-5) Flowchart of interval encoding / decoding

[0120] Figure 5A A flowchart showing an example of interval encoding. Figure 5B A flowchart showing an example of interval decoding. First, refer to Figure 5A Describes the order of interval encoding.

[0121] Based on the input bit history, the encoder 301 refers to the probability table 304 for the probability value of the next bit being "0" (501). Based on this probability value, the encoder 301 divides the numerical axis interval (the interval to be divided) into two segments (502). During the division, the interval size is multiplied by the probability value. This is the part that takes the most time in the encoding process. Furthermore, the encoder 301 selects one of the two segments based on whether the input bit value is "0" or "1" (503).

[0122] Next, in step 504 , the encoder 301 determines whether the bit input is completed. If the bit input is completed ( 504 : Yes), the process moves to step 506 . If there is still input ( 504 : No), the process moves to step 505 .

[0123] In step 505, encoder 301 updates the probability values ​​used in probability table 304 and updates the bit history for the next bit to be encoded. The probability value is updated by increasing it if the input bit value is "0" and decreasing it if it is "1." For example, if the next input bit after "11" is "0," the bit history is updated to "110." Encoder 301 then returns to step 501 and continues encoding.

[0124] On the other hand, in step 506, the encoder 301 determines the coordinate value of the segment to be retained last, for example, the value with the least number of bits among the values ​​included in the segment, and outputs it as a code, thereby completing the encoding process.

[0125] Reference Figure 5B The following describes the order of an example of interval decoding. The decoder 311 refers to the probability table 314 based on the output bit history and obtains the probability value of the next bit being "0" (511). The decoder 311 divides the numerical axis interval into two segments based on its probability value (512). When dividing, the interval size is multiplied by the probability value. This is the part that takes the most time in the decoding process. Next, the decoder 311 selects the segment containing the value of the input code from the two segments (513). The decoder 311 outputs the bit value "0" or "1" represented by the selected segment (514).

[0126] Next, in step 515 , the decoder 311 determines whether the output of bits has been completed. If so ( 515 : Yes), the decoding process ends. If there is still output ( 515 : No), the process moves to step 516 .

[0127] In step 516, decoder 311 updates the probability value used in probability table 314 and updates the bit history for decoding the next bit. The probability value is updated by increasing it if the output bit value is "0" and decreasing it if it is "1." For example, if the next output bit of "11" is "0," the bit history is updated to "110." Decoder 311 then returns to step 511 and continues decoding.

[0128] In addition, in the above-mentioned encoding, the probability table is updated every time 1 bit is processed on the bit string (symbol stream) given by encoding and decoding. The longer the processed bit string, the more suitable the probability distribution for the bit string is, and the higher the compression efficiency (that is, the further behind the processed bit string is, the higher the possibility of a high compression effect).

[0129] In addition to this adaptive updating of the probability table, it is also possible to examine the entire bitstream before starting encoding to create a probability table, and then use this probability table as a fixed parameter for encoding and decoding. Compared to the above adaptive method, when the probability table is created and used independently of the encoding and decoding processes, since the probabilities (segment divisions) are independent of the position in the processed bitstream, encoding and decoding can be performed from positions other than the beginning. However, in this case, encoder 301 and decoder 311 need to share the same probability table.

[0130] (3) High-speed section coding processing

[0131] Figure 6 This is a functional block diagram showing an example of a method for speeding up section encoding processing. Figure 1 The LZMA compression and decompression circuit 104 performs the compression process shown in the block diagram. Figures 3A to 5B In the example described above, the multiplication process is performed with reference to the probability for each input bit. Therefore, only one bit can be processed in one operation cycle, which may cause the processing performance of the LZMA algorithm to be slow.

[0132] Figure 6 The illustrated interval coding function 600 accelerates the interval coding process by operating multiple encoders simultaneously. Specifically, N intervals are prepared (N>1) and N types of bit histories are pre-prepared from the input bit stream. This function simultaneously references N probability values ​​in a probability table and performs multiplication processing on the N input bits in parallel, thereby increasing coding performance by a factor of N.

[0133] Figure 6 The following shows an example of encoding based on this high-speed technology when N=4. The four encoders 601A to 601D are all connected to Figure 3AThe encoder 301 performs the same process. Each encoder obtains and uses the probability values ​​one by one from the probability table 604. These four probability values ​​are referenced using the bit histories 605A to 605D as indices.

[0134] Bit history 605A is used by encoder 601A when processing the first bit "1" of input bit sequence 602 and has a value of NULL. Bit history 605B is used by encoder 601B when processing the second bit "1" of input bit sequence 602 and has a value of "1".

[0135] Bit history 605C is used when encoder 601C processes the third bit "0" of input bit sequence 602 and has a value of "11". Bit history 605D is used when encoder 601D processes the fourth bit "1" of input bit sequence 602 and has a value of "110".

[0136] Generally speaking, the bit history used to encode the Nth bit is formed by concatenating the 1st to (N-1)th bits. By preparing four types of bit histories in this way, the four encoders 601A to 601D can simultaneously refer to the four probability values ​​from the probability table 604 and perform multiplication using these probability values ​​simultaneously.

[0137] The four subcodes 603A to 603D output from encoders 601A to 601D are concatenated to form output code 606. Output code 606 corresponds to the compressed data of the LZMA algorithm. This method can process a 4-bit input in a single computation cycle, thus improving the performance of interval coding in the LZMA algorithm compression process by four times.

[0138] Figure 7 Indicates reference Figure 6 This is a flowchart example of a method for increasing the speed of interval coding. Figure 7 The flowchart illustrates the sequence of the high-speed interval coding method. First, the LZMA compression and decompression circuit 104 creates N types of bit histories (701) used to encode the N bits in the input bit string. N is an integer greater than or equal to 2. The N encoders obtain the probability value of the next bit being "0" from the probability table based on each bit history (702). Based on these probability values, the N encoders divide the N value axis intervals (division target intervals) into two segments (703).

[0139] The multiplication of the interval size and the probability value by the N encoders is performed in parallel. In the first cycle, the value axis interval (interval size) is common to the N encoders. Figure 6In the example shown, it is [0, 1]. After the second cycle, the segment selected in the previous cycle becomes the target value axis interval (interval size). Each encoder selects one segment (704) from the left and right segments based on whether the input bit value is "0" or "1".

[0140] Next, in step 705, the LZMA compression / decompression circuit 104 determines whether the bit input has ended. If the bit input has ended (705: Yes), the process moves to step 707. If there is still bit input (705: No), the process moves to step 706. In step 706, the LZMA compression / decompression circuit 104 updates the N probability values ​​used in the probability table. The probability values ​​are updated by incrementing if the input bit value is "0" and decrementing if it is "1."

[0141] The LZMA compression / decompression circuit 104 then returns to step 701 to continue encoding. For example, if N is 4 and 8-bit character encoding is performed, the first 4 bits are encoded in the first cycle of this process, and the second 4 bits are encoded in the second cycle. In the second cycle, the bit history used to encode the fifth bit is the first 4 bits.

[0142] For example, when the input bit string is 6 bits, the first half, 4 bits or 3 bits, may be encoded in the first cycle, and the second half, 2 bits or 3 bits, may be encoded in the second cycle. The maximum value of the input bit string to the LZMA compression / decompression circuit 104 is 4, and bit strings less than this value can be encoded.

[0143] In step 707, each encoder generates coordinate values ​​for the segment to be retained last, for example, the value with the fewest bits among the values ​​contained in the segment. The LZMA compression / decompression circuit 104 outputs the N concatenated bit strings as a code, and the encoding process ends.

[0144] exist Figure 6 In the example of interval coding shown, the integer N>1, the number of input bits is N bits, that is, the maximum length of the bit history is (N-1) bits. This interval coding uses N intervals of the divided object, which can increase the coding speed by N times. Figure 7 As mentioned in the example of 8-bit text characters in the description, when the integer M>N and the number of input bits is M bits, that is, the maximum length of the bit history is (M-1) bits, the N intervals of the divided object can be used to speed up the encoding of the input bits.

[0145] The following describes this method using the case of M = 8 and N = 4 as an example. LZMA compression / decompression circuit 104 includes a 255-entry probability table indexed by bit histories up to 7 bits. LZMA compression / decompression circuit 104 prepares four types of bit histories (null, 1 bit, 2 bits, and 3 bits) used to encode the first 4 bits of an 8-bit input bit string and simultaneously references the four corresponding probability values ​​in the probability table. Using these probability values, LZMA compression / decompression circuit 104 encodes the first 4 bits in parallel during the first cycle.

[0146] Next, LZMA compression / decompression circuit 104 prepares four types of bit histories (4 bits, 5 bits, 6 bits, and 7 bits, including the first 4 bits at the beginning) for encoding the last 4 bits of the input bits, and simultaneously references the corresponding four probability values ​​from the probability table. Using these probability values, LZMA compression / decompression circuit 104 encodes the last 4 bits in parallel in the second cycle.

[0147] Thus, the LZMA compression / decompression circuit 104 receives 8 bits as input, processes them in two cycles (ie, quadruple the performance), generates four subcodes, and concatenates them to form an output code.

[0148] Generally speaking, when encoding an M-bit interval code input, a probability table with (2^M-1) entries indexed by the maximum (M-1)-bit bit history and N encoders is used, processing the M bits in [M / N] operation cycles, thereby improving performance. Alternatively, LZMA compression / decompression circuit 104 can generate subcodes without performing the aforementioned parallel processing.

[0149] (4) Speeding up section decoding processing

[0150] The speed of the interval decoding process is increased by a factor of N simply by making N Figure 3B It is not possible to operate decoders 311 in parallel. This is because the bit history used in the processing of decoder X is not determined until decoder Y, which decoded the previous bit, outputs the processing result. Therefore, decoder X and decoder Y cannot simultaneously refer to the probability table and perform multiplication using the probability values, making parallelization impossible.

[0151] Hereinafter, a method for increasing the speed of section decoding processing according to one embodiment of this specification will be described. Figure 8 This is a functional block diagram showing a method for speeding up section decoding processing. Figure 1 The LZMA compression and decompression circuit 104 performs decompression processing through the interval decoding function 800 shown in the block diagram.

[0152] Figure 8This shows an example of interval decoding when N=4, that is, the number of bits in the output bit string is 4. The 15 decoders 8A (1), 8B0 and 8B1 (2), 8C00 to 8C11 (4), and 8D000 to 8D111 (8) are all the same. Figure 3B The decoder 311 performs the same process. Figure 8 The four subcodes 803A to 803D input to the 15 decoders are separated from the input code 802 (equivalent to the compressed data of the LZMA algorithm) and Figure 6 The four subcodes 603A to 603D are the same. In addition, one subcode may be shared by multiple decoders.

[0153] Each decoder obtains and uses probability values ​​one by one from probability table 804 to output candidate bit values. These 15 probability values ​​are values ​​referenced using all possible bit histories as indices.

[0154] The bit history value used by one decoder 8A for decoding the first bit of output bit string 806 is NULL. The bit history values ​​used by two decoders 8B0 and 8B1 for decoding the second bit of output bit string 806 are "0" and "1", respectively.

[0155] The bit history values ​​used by the four decoders 8C00 to 8C11 for decoding the third bit of output bit sequence 806 are "00," "01," "10," and "11," respectively. The bit history values ​​used by the eight decoders 8D000 to 8D111 for decoding the fourth bit of output bit sequence 806 are "000," "001," "010," "011," "100," "101," "110," and "111," respectively.

[0156] Generally speaking, the number of bit histories used in decoding the Kth bit is 2^(K-1). Each bit history is a bit pattern (bit sequence) of (K-1) bits that can potentially be bits 1 through (K-1) of output bit sequence 806. By preparing 15 different bit histories, 15 probability values ​​are simultaneously referenced from probability table 804, and 15 decoders simultaneously perform multiplication using these probability values.

[0157] If the first bit of output bit string 806 from decoder 8A is "1," it is clear that the second bit output by decoder 8B1, which decoded both decoders 8B0 and 8B1 assuming the first bit was "1," is the correct result. Therefore, selector 805B selects the "1" output by decoder 8B1, out of the two second-bit candidates output by decoders 8B0 and 8B1. In other words, the first and second bits are determined to be "11."

[0158] This shows that the third bit output by decoder 8C11, which decoded the 1st and 2nd bits as "11," is the correct result. Therefore, selector 805C selects "0" output by decoder 8C11 from the four third-bit candidates output by decoders 8C00 to 8C11. In other words, bits 1 through 3 are determined to be "110."

[0159] This shows that the fourth bit output by decoder 8D110, which decoded the first through third bits assuming they were "110," is the correct result. Therefore, selector 805D selects "1," output by decoder 8D110, from the eight fourth-bit candidates output by decoders 8D000 through 8D111.

[0160] The above determines that the four bits of output bit string 806 are "1101." Generally speaking, LZMA compression / decompression circuit 104 includes 2^(K-1) decoders to decode the K-th bit, and stores the 2^(K-1) K-th bit candidates output by these decoders. LZMA compression / decompression circuit 104 selects the candidate output by a decoder that decoded the previously determined values ​​of bits 1 to (K-1) as the K-th bit.

[0161] The bit selection process performed by selectors 805B-805D takes significantly less time than the decoder's multiplication process. Therefore, this method can process a 4-bit output in a single computation cycle. Consequently, the performance of the interval decoding process in the LZMA algorithm's decompression process is quadrupled compared to conventional methods.

[0162] The following reference Figure 9 Indicates reference Figure 8 The procedure for speeding up the interval decoding described above. First, the LZMA compression / decompression circuit 104 creates (2^N-1) bit histories (901) that can be used to decode N bits in the output bit string. The bit histories used to decode the Kth bit are 2^(K-1).

[0163] The (2^N-1) decoders obtain the probability value (902) that the next bit is "0" from the probability table 804 based on the bit history they each receive and hold, and divide the numerical axis interval (division object interval) into two segments (903) based on the probability value. The numerical axis intervals divided by the 2^(K-1) decoders used in the decoding of the Kth bit are the same. Specifically, in the first cycle, the numerical axis intervals of all decoders are the same. Figure 8In the example, it is [0, 1). After the second cycle, the value axis interval of the K-th bit decoder is the interval of the segmentation result obtained by the decoder that output the correct answer for the K-th bit in the previous cycle. When performing segmentation, the interval size is multiplied by the probability value.

[0164] The decoder selects the segment (904) of the two segments that contains the value of the input subcode, and generates a bit value "0" or "1" represented by the selected segment (905). In addition, the number of bit values ​​generated is (2^N-1), and the number of candidates for the K-th bit is 2^(K-1). Then, the selector selects one correct bit from each candidate in sequence, starting from the first bit, and determines the N-bit pattern and outputs it (906). In addition, in selecting the correct answer value of the K-th bit, the correct answer values ​​of the 1st to (K-1)th bits are used as the bit history.

[0165] Next, in step 907 , the LZMA compression / decompression circuit 104 determines whether the output of bits has ended. If so ( 907 : Yes), the decoding process ends. If there is still output ( 907 : No), the process moves to step 908 .

[0166] In step 908, the LZMA compression / decompression circuit 104 updates the N probability values ​​used in probability table 804. The probability values ​​are updated by increasing them if the output bit value is "0" and decreasing them if it is "1." Furthermore, the LZMA compression / decompression circuit 104 uses the segment selected in step 904 by the decoder that output the correct bit value as the value axis interval for the next cycle. The segment selected in step 904 by the one decoder that output the correct bit value for the Kth bit among the 2^(K-1) decoders for the Kth bit is used as the value axis interval divided in step 903 for the next Kth bit decoding.

[0167] The LZMA compression / decompression circuit 104 then returns to step 901 to continue decoding. For example, if the first 4 bits and the second 4 bits of an 8-bit character are encoded in two cycles, the bit history used to decode the fifth bit in the second cycle of this process is the bit string of the first 4 bits.

[0168] For example, when 6-bit input data is divided into the first 4 bits and the remaining 2 bits and encoded in two cycles, the LZMA compression / decompression circuit 104 can decode 4 or 3 bits in the first cycle and then 2 or 3 bits in the second cycle. The maximum input bit string to the LZMA compression / decompression circuit 104 is 4, and bit strings below this value can be decoded.

[0169] exist Figure 8In the high-speed decoding method of the interval code shown, when the integer N>1, the number of input bits is N bits (that is, the length of the bit history is a maximum of (N-1) bits), and N segmentation target intervals are used, the speed is increased to N times. Figure 9 As mentioned in the example of 8-bit characters, when the integer M>N and the number of input bits is M, that is, the length of the bit history is a maximum of (M-1) bits, the decoding speed of the output bit string can be increased using N segmentation target intervals.

[0170] The following describes an example of decoding an 8-bit output bit string. Figure 8 It also has 15 decoders, Figure 8 Similarly, the 15 decoders are fed with 4 subcodes separated from the input code (corresponding to compressed data using the LZMA algorithm). Each decoder retrieves and uses probability values ​​one by one from a 255-entry probability table indexed by a bit history of up to 7 bits.

[0171] The 15 probability values ​​referenced in the first cycle are values ​​that use as indices all bit histories (null, 1 bit, 2 bits, and 3 bits, respectively) that could potentially be the first 4 bits of the 8-bit output bit string. The bit history used by the decoder that decodes the first bit of the output bit string is NULL.

[0172] The two decoders that decode the second bit of the output bit string use bit history values ​​of "0" and "1," respectively. The four decoders that decode the third bit of the output bit string use bit history values ​​of "00," "01," "10," and "11," respectively. The eight decoders that decode the fourth bit of the output bit string use bit history values ​​of "000," "001," "010," "011," "100," "101," "110," and "111," respectively.

[0173] The 15 decoders perform multiplication in parallel using the probability values ​​referenced by these bit histories. Figure 8 Similarly, the bit selection process of the selector determines the values ​​of bits 1 to 4 in sequence. Here, for example, "1101"

[0174] Next, the 15 probability values ​​referenced in the second cycle are values ​​referenced using all the bit histories that may be the second half of the 4 bits of the 8-bit output bit string (respectively, the 4-bit, 5-bit, 6-bit, and 7-bit "1101" determined in the first cycle contained in the header) as indexes.

[0175] The bit history value used by the one decoder that decodes the fifth bit of the output bit string is "1101." The bit history values ​​used by the two decoders that decode the sixth bit of the output bit string are "11010" and "11011," respectively. The bit history values ​​used by the four decoders that decode the seventh bit of the output bit string are "110100," "110101," "110110," and "110111," respectively.

[0176] The values ​​of the bit histories used by the eight decoders that decode the eighth bit of the output bit string are "1101000", "1101001", "1101010", "1101011", "1101100", "1101101", "1101110", and "1101111", respectively.

[0177] The 15 decoders perform multiplication in parallel using the probability values ​​referenced by these bit histories. Figure 8 Likewise, the bit selection process of the selector sequentially determines the values ​​of the 5th bit to the 8th bit.

[0178] and Figure 8 Similarly, the bit selection process of the selector is performed in a significantly shorter time than the multiplication process of the decoder. Therefore, this method can process an 8-bit output in two calculation cycles. As described above, in the probability table of 255 entries, 15 entries are referenced in the first cycle, and in the second cycle, 15 entries are selected from the remaining 240 entries for reference. In the second cycle, by indexing using the bit history including the first half of the 4 bits determined in the first cycle, the number of entries referenced is reduced to 1 / 16.

[0179] If generalized, the decoding process of the interval code that outputs M bits can improve its performance by using a probability table with (2^M-1) entries indexed by the bit history of the maximum (M-1) bits and (2^N-1) decoders to process the M bits in [M / N] operation cycles.

[0180] As described above, one embodiment of the present specification can decompress data compressed using interval coding at high speed, thereby improving the read response performance of compressed data in, for example, a device storage system having a data compression function based on an interval coding algorithm.

[0181] (5) Post-processing compression

[0182] The following describes post-processing compression, which can improve the compression rate of data stored in the SSD 105 while suppressing the impact on write access from the host to the storage system. Figure 10 It is a diagram used to illustrate the logical structure of post-processing compression. Figure 11It is a flowchart used to illustrate post-processing compression.

[0183] Reference Figure 10 In the inline compression, the LZMA compression and decompression circuit 104 compresses the write data 211 from the host and stores the inline compressed data in the SSD 105. In the post-processing compression, the inline compressed data stored in the SSD 105 by the LZMA compression and decompression circuit 104 is decompressed, and the CPU 107 recompresses the decompressed data and stores it in the SSD 105.

[0184] After dictionary compression 171, CPU 107 executes interval encoding 172 that is interchangeable with interval encoding 203 in LZMA compression and decompression circuit 104 (using the same compression and decompression algorithm). Dictionary compression 171 (second dictionary compression) performed by CPU 107 has a higher ability to search for identical character strings than dictionary compression 202 (first dictionary compression) performed by LZMA compression and decompression circuit 104. This allows for a higher compression rate for post-processing compression than inline compression. Furthermore, since interval encoding 172 performed by CPU 107 is interchangeable with interval encoding 203 in LZMA compression and decompression circuit 104, decompression by LZMA compression and decompression circuit 104 can be used during reading, thereby minimizing degradation in reading performance.

[0185] Furthermore, interval coding 203 and 172 may be omitted, or coding based on other compression / decompression algorithms may be performed. Interval coding may also be performed in a process separate from the aforementioned parallel processing. By performing dictionary compression and coding other than dictionary compression, such as interval coding or entropy coding such as Huffman coding, the data compression rate can be improved.

[0186] Reference Figure 11 In step 1001, CPU 107 compares its current operating rate with a preset threshold to determine whether the operating rate is less than the threshold. If the operating rate is above the threshold (1001: No), the process ends. If the operating rate is less than the threshold (1001: Yes), the process proceeds to step 1002.

[0187] By starting post-processing compression when the CPU 107 load falls below a threshold, the impact on other processes can be minimized. Alternatively, a value representing CPU load, such as the number of executed tasks, other than the CPU operating rate, may be referenced. Alternatively, post-processing compression may be performed periodically, without reference to CPU load.

[0188] For example, CPU 107 can also perform post-processing compression in conjunction with garbage collection. CPU 107 appends updated data at a certain address within a volume to a new address in SSD 105. Old data stored at the aforementioned address in SSD 105 becomes invalid data. Garbage collection saves valid data in SSD 105 to the new address area and converts the invalid data area into a free area. While moving valid data, CPU 107 performs post-processing compression on the target data read from SSD 105 and stores it at the new address in SSD 105. This allows for efficient post-processing compression.

[0189] In step 1002 , CPU 107 selects and reads one of the inline compressed data from SSD 105 and stores it in buffer memory 106 . As described above, in inline compression, the LZMA compression / decompression circuit 104 compresses the write data from the host and stores it in SSD 105 .

[0190] Data stored in SSD 105 that has not undergone post-processing compression can be managed using management information (not shown). For example, the management information may include an address within the volume, information about uncompressed data, and the address of SSD 105 storing valid data at that address. The management information may be stored in memory within storage controller 103.

[0191] Management information may also include information about the time when the data was saved (updated), along with information about whether or not post-processing compression was performed on the valid saved data. CPU 107 may select data to be post-processed compressed based on the time the data was updated. For example, CPU 107 may prioritize post-processing compression on data with an older update time among candidate data for post-processing compression. Candidate data is valid data that has not been post-processed compressed. CPU 107 may select data to be post-processed compressed from the oldest data or from data whose time since update exceeds a threshold.

[0192] In step 1003, the LZMA compression / decompression circuit 104 decompresses the inline compressed data stored in the buffer memory 106 according to the instruction from the CPU 107 and stores the decompressed data in the buffer memory 106. By using the LZMA compression / decompression circuit 104, data can be decompressed efficiently in a short time.

[0193] In step 1004, CPU 107 performs dictionary compression 171, which enhances string search capabilities compared to inline compression. This allows for a higher compression ratio than inline compression. For example, CPU 107 may perform dictionary compression using a hash table with a larger upper limit on the number of hash bits or entries. Alternatively, CPU 107 may perform dictionary compression using a hash table with a smaller number of characters in the string.

[0194] The hash table may have both a larger number of hash bits and an upper limit on the number of entries than the value in inline compression. Alternatively, the number of hash bits and / or the upper limit on the number of entries may be larger than the value in inline compression, resulting in a smaller number of characters in the string. The hash table may also be stored in memory within the storage controller 103.

[0195] In step 1005, CPU 107 encodes the data into a plurality of subcodes by executing interval encoding 172 interchangeable with interval encoding 203 of LZMA compression / decompression circuit 104. This enables high-speed processing using LZMA compression / decompression circuit 104 when reading compressed data.

[0196] In step 1006, CPU 107 outputs the code (post-processed compressed data (in Figure 11 The post-compressed data (hereinafter referred to as the post-compressed data) is stored in buffer memory 106. In step 1007, CPU 107 writes the post-processed compressed data to an address area in SSD 105 that is different from the original inline compressed data. In step 1008, the selected inline compressed data is invalidated and the post-processed compressed data is validated. Specifically, CPU 107 updates the management information that manages the address of SSD 105 and the validity / invalidity of the stored data.

[0197] Alternatively, in step S1006, the post-processed compressed data may be saved in a buffer memory, and then, in parallel with the action of writing the post-processed compressed data on the buffer memory to SSD 105, the data may be transferred and saved to a storage area on the cloud for backup.

[0198] In this case, backup to the cloud is limited when writing inline compressed data to SSD 105. This allows for a hybrid environment combining on-premises and cloud systems, while reducing cloud capacity utilization and achieving enhanced data preservation. Furthermore, data transfer to the cloud can be done on a schedule other than the one described above. Alternatively, the entire system can be configured as a hybrid cloud system to periodically transfer post-processed compressed data.

[0199] Furthermore, regarding the backup function using cloud storage capacity, for example, a configuration can be employed in which the processing after S1006 (saving the post-processed compressed data) is applied only to data storage on the cloud. That is, in S1006, the post-processed compressed data is saved in the buffer memory, and then the post-processed compressed data is transferred to the cloud storage area.

[0200] At this time, the selection of data to be compressed after post-processing can be based on, for example, the recent usage frequency of the inline compressed data. For example, the usage frequency can be monitored, and data with a usage frequency lower than a preset frequency (i.e., low usage frequency) can be identified as data to be compressed after post-processing. Such data can be transmitted to the cloud after post-processing compression, and the corresponding inline compressed data can be invalidated according to the process of S1008.

[0201] This hybrid system allows for the placement of frequently used data in the local environment to ensure responsiveness, while also placing less frequently used data in the cloud environment with high compression. In other words, it achieves a balance between responsiveness in data reading and writing and efficient use of storage space.

[0202] [Example 2]

[0203] Another embodiment of post-processing compression will be described. Below, the differences from Embodiment 1 will be mainly described. The description of Embodiment 1 can be applied to the same configuration as that of Embodiment 1. Figure 12 This is a logical diagram illustrating another embodiment of post-processing compression. Figure 13 is a flowchart illustrating another embodiment of post-processing compression.

[0204] Reference Figure 12 In this embodiment, post-processing compression compresses data dictionary-compressed by CPU 107 through interval code 203 of LZMA compression and decompression circuit 1104. This reduces the load on CPU 107 and speeds up the compression process. LZMA compression and decompression circuit 1104, in addition to the components of LZMA compression and decompression circuit 104 in Example 1, also includes selector 209. Selector 209 selects dictionary compression 202 for inline compression and selects bit string 215 resulting from dictionary compression performed by CPU 107 for post-processing compression, outputting it to interval code 203.

[0205] Reference Figure 13 , steps 1201 to 1204 are the same as Figure 11 The steps 1001 to 1004 in the flowchart are the same. In step 1205, the CPU 107 compresses the data which has been dictionary compressed by the CPU 107 by using the interval code 203 of the LZMA compression and decompression circuit 1104. Steps 1206 to 1208 are the same as those in the flowchart. Figure 11Steps 1006 to 1008 in the flowchart of FIG. 1 are the same.

[0206] Alternatively, the post-processed compressed data may be saved in a buffer memory in step S1206, and then, in parallel with the action of writing the post-processed compressed data on the buffer memory to SSD 105, the data may be transferred and saved to a storage area on the cloud for backup.

[0207] In this case, backup to the cloud is limited when writing inline compressed data to SSD 105. This allows for a hybrid environment combining on-premises and cloud systems, while reducing cloud capacity utilization and achieving enhanced data preservation. Furthermore, data transfer to the cloud can be done on a schedule other than the one described above. Alternatively, the entire system can be configured as a hybrid cloud system to periodically transfer post-processed compressed data.

[0208] Furthermore, regarding the backup function using cloud storage capacity, for example, a configuration can be employed in which the processing after S1206 (saving the post-processed compressed data) is applied only to data storage on the cloud. That is, in S1206, the post-processed compressed data is saved in the buffer memory, and then the post-processed compressed data is transferred to the cloud storage area.

[0209] At this time, the data to be subjected to post-processing compression may also be selected based on, for example, the recent usage frequency associated with the inline compressed data. For example, the usage frequency may be monitored, and data with a usage frequency lower than a preset frequency (i.e., low usage frequency) may be determined as data to be subjected to post-processing compression. Such data may be transmitted to the cloud after post-processing compression, and the corresponding inline compressed data may be invalidated according to the process of S1208.

[0210] This hybrid system allows for the placement of frequently used data in the local environment to ensure responsiveness, while also placing less frequently used data in the cloud environment with high compression. In other words, it achieves a balance between responsiveness in data reading and writing and efficient use of storage space.

[0211] Furthermore, the storage systems of Embodiments 1 and 2 and their variations can achieve resource saving by reducing the number of storage drives due to reduced storage capacity by reducing the amount of data, thereby reducing power consumption of the storage drives.

[0212] Furthermore, the present invention is not limited to the above-described embodiments and includes various variations. For example, the above-described embodiments are described in detail to facilitate understanding of the present invention and are not limited to having all of the described configurations. Furthermore, a portion of the configuration of a particular embodiment may be replaced with a configuration of another embodiment, and a configuration of another embodiment may be added to a configuration of a particular embodiment. Furthermore, with respect to a portion of the configuration of each embodiment, other configurations may be added, deleted, or replaced.

[0213] Furthermore, the aforementioned components, functions, and processing units may be implemented in hardware, for example, by designing some or all of them using integrated circuits. Furthermore, the aforementioned components and functions may be implemented in software by having a processor interpret and execute programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions may be stored in a storage device such as a memory, a hard disk, or an SSD, or a storage medium such as an IC card or an SD card.

[0214] In addition, regarding control lines and information lines, only those considered necessary for explanation are shown, and not all control lines or information lines are necessarily shown on the product. In fact, it can be assumed that almost all components are connected to each other.

Claims

1. A storage system, wherein: The controller includes a processor and a data compression and decompression circuit. The controller performs inline compression of plaintext data from the host and post-processing compression of the inline compressed data stored in one or more storage drives. The above inline compression includes: The data compression and decompression circuit performs compression processing including first dictionary compression on the plain text data from the host to generate inline compressed data; saving the inline compressed data to the one or more storage drives mentioned above, The above post-processing compression includes: Decompressing the inline compressed data read from the one or more storage drives by the data compression and decompression circuit to generate the plaintext data; The plaintext data is subjected to compression processing including second dictionary compression by the processor to generate post-processed compressed data, which is stored in the one or more storage drives. The second dictionary compression has a better character string search capability than the first dictionary compression.

2. The storage system according to claim 1, wherein: The compression process performed by the data compression and decompression circuit includes, after executing the first dictionary compression, encoding based on a first compression and decompression algorithm different from the first dictionary compression. In the post-processing compression, after the processor compresses the plaintext data using the second dictionary compression, encoding based on the first compression and decompression algorithm is performed to generate the post-processing compressed data.

3. The storage system according to claim 2, wherein: In the post-processing compression, the processor performs encoding based on the first compression and decompression algorithm.

4. The storage system according to claim 2, wherein: In the post-process compression, the data compressed by the second dictionary compression is input to the data compression and decompression circuit, and encoding is performed based on the first compression and decompression algorithm.

5. The storage system according to claim 1, wherein: The controller starts the post-processing compression when the operating rate of the processor is less than a preset threshold. The storage system according to claim 1 , wherein: In the post-process compression, the oldest data is selected from the candidate data stored in the one or more storage drives.

7. The storage system according to claim 1, wherein: The second dictionary-compressed hash table has at least one of a hash bit number and an upper limit on the number of entries greater than those of the first dictionary-compressed hash table, and a character string in the second dictionary-compressed hash table has fewer characters than those in the first dictionary-compressed hash table.

8. The storage system according to claim 1, wherein: The controller performs the post-processing compression during garbage collection of the one or more storage drives.

9. The storage system according to claim 2, wherein: The controller reads the compressed data obtained by the post-processing compression from the one or more storage drives, decompresses the compressed data into the plain text data through the data compression and decompression circuit, and sends the plain text data to the host.

10. A data compression method is a data compression method in a storage system, wherein: The above storage system includes a processor and a data compression and decompression circuit. In the above data compression method, Includes inline compression of plaintext data from the host, and post-processing compression of inline compressed data stored in one or more storage drives, The above inline compression includes: The data compression and decompression circuit performs compression processing including first dictionary compression on the plain text data from the host to generate inline compressed data; saving the inline compressed data to the one or more storage drives mentioned above, The above post-processing compression includes: Decompressing the inline compressed data read from the one or more storage drives by the data compression and decompression circuit to generate the plaintext data; The plaintext data is subjected to compression processing including second dictionary compression by the processor to generate post-processed compressed data, which is stored in the one or more storage drives. The second dictionary compression has a better character string search capability than the first dictionary compression.

Images