Storage system

Abstract

To improve a compression rate of a data.SOLUTION: A controller performs an inline compression of plaintext data from a host and a post-process compression of inline compressed data stored in one or more storage drives. In the inline compression, a data compression / decompression circuit performs compression processing including a first dictionary compression on the plaintext data from the host to generate the inline compressed data, and stores the inline compressed data in one or more storage drives. In the post-process compression, the inline compressed data read from the one or more storage drives is decompressed by the data compression / decompression circuit to generate plaintext data. In the post-process compression, the compression processing including a second dictionary compression by a processor having a character string search capability superior to that of the first dictionary compression is performed on the plaintext data to generate the post-process compressed data, and the post-process compressed data is stored in the one or more storage drives.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to data compression technology in a storage system.

Background Art

[0002] A storage system, which is an information device for accumulating and managing a large amount of data, can reduce the cost per unit of capacity by storing more data. Therefore, some storage systems are equipped with a function of compressing the written data and then storing it in a disk drive.

[0003] For example, in recent years, as a storage medium of a storage system, in addition to or instead of a HDD (Hard Disk Drive), an SSD (Solid State Drive) equipped with a NAND-type flash memory, which is a non-volatile semiconductor memory, has been increasingly adopted. Since an SSD does not have a physical head seek mechanism like a HDD in data access, it has less head start delay (latency) and excellent response performance in random data reads.

[0004] Therefore, in applications such as databases that require high-speed random reads, the replacement of the storage medium of the storage system from HDD to SSD is progressing. However, although the bit cost of SSD has been decreasing year by year with the high integration of flash memory cells, it still remains higher than that of HDD.

[0005] Therefore, many storage systems using SSD as a storage medium introduce reversible compression technology and have a function of reducing the data size stored in the SSD. Thereby, the storage capacity of the system can be virtually increased, the cost per unit of capacity can be reduced, and it can be made closer to that of a storage system using HDD as a storage medium.

[0006] Patent Document 1 serves as background technology for this disclosure. Patent Document 1 discloses a storage system that uses the LZMA algorithm. In the LZMA compression process, the plaintext data before compression is first subjected to dictionary compression. The dictionary compression result is then subjected to range coding. This generates LZMA compressed data. In the LZMA decompression process, the compressed data is first subjected to range decoding. The decoding result is then subjected to plaintext decompression. This generates the original plaintext data. Patent Document 1 speeds up inline compression and decompression by using a multi-range configuration for the LZMA range code and parallel processing of the range coder and decoder. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2022-095015 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Inline compression, which compresses data received from the host and stores it on a storage drive, requires both a high compression ratio and the ability to meet the I / O performance requirements of the storage system. Data compression using hardware circuits (accelerators) enables high-speed data compression, thus meeting even higher I / O performance requirements. However, storage system design limitations are imposed on the logical size of the FPGA (Field Programmable Gate Array) of the accelerator. .death Therefore, the I / O performance requirements of the storage system and the circuitry for data compression Model There is a need for technology that can improve the compression ratio of stored data while meeting the requirements of the limitations. [Means for solving the problem]

[0009] A storage system according to one aspect of the present invention includes a processor and a controller including a data compression / decompression circuit, wherein the controller performs inline compression of plaintext data from a host and post-process compression of inline compressed data stored in one or more storage drives, the inline compression includes the data compression / decompression circuit performing a compression process including a first dictionary compression on plaintext data from a host to generate inline compressed data and storing the inline compressed data in the one or more storage drives, the post-process compression includes the data compression / decompression circuit decompressing the inline compressed data read from the one or more storage drives to generate plaintext data, and the processor performing a compression process including a second dictionary compression having better string search capability than the first dictionary compression on the plaintext data to generate post-process compressed data and storing it in the one or more storage drives. [Effects of the Invention]

[0010] According to one aspect of the present invention, data stored in a storage system can be compressed more effectively. [Brief explanation of the drawing]

[0011] [Figure 1] An example of a storage system configuration is shown. [Figure 2A] This section outlines the LZMA algorithm. [Figure 2B] Here is a concrete example of dictionary compression. [Figure 2C] An example of a hash table configuration is shown. [Figure 3A] The functional block diagram for range coding is shown. [Figure 3B] A functional block diagram of range decoding is shown. [Figure 4A] An example is given to illustrate the principle of range coding. [Figure 4B] Here are some other examples to illustrate the principle of range coding. [Figure 5A] A flowchart of the range coding process is shown. [Figure 5B] Shows a flowchart of range decoding processing. [Figure 6] Shows a functional block diagram of a method for accelerating range encoding processing. [Figure 7] Shows a flowchart of a method for accelerating range encoding processing. [Figure 8] Shows a functional block diagram of a method for accelerating range decoding processing. [Figure 9] Shows a flowchart of a method for accelerating range decoding processing. [Figure 10] It is a logical configuration diagram for explaining post - process compression. [Figure 11] It is a flowchart for explaining post - process compression. [Figure 12] It is a logical configuration diagram for explaining another example of post - process compression. [Figure 13] It is a flowchart for explaining another example of post - process compression.

Modes for Carrying Out the Invention

[0012] In the following, for convenience, when necessary, it will be divided into multiple sections or examples for explanation. However, unless otherwise explicitly stated, they are not unrelated to each other. One is related to a partial or total modification example, details, supplementary explanation, etc. of the other. Also, in the following, when referring to the number of elements, etc. (including the number, numerical value, quantity, range, etc.), unless otherwise explicitly stated or limited to a specific number in principle, it is not limited to that specific number, and it may be more than or less than the specific number.

[0013] One embodiment of this specification compresses data written to a storage system in an inline process using an accelerator (logic circuit) and writes it to the storage drive. Furthermore, the inline compressed data written to the storage drive is converted back to plaintext in a post-process, recompressed using a compression method that enhances dictionary compression from the accelerator, and written back to the storage drive. This makes it possible to improve the data compression ratio of the storage system (making the stored data even smaller) without increasing the circuit size of the accelerator.

[0014] LZMA compression can improve compression ratios by increasing the search capability of the dictionary. However, LZMA compression using hardware circuits (accelerators) increases the logical size of the FPGA (Field Programmable Gate Array) by increasing the search capability of the dictionary. Furthermore, software-based dictionary compression in inline compression can cause a decrease in I / O performance.

[0015] From the perspective of storage system I / O performance requirements and the FPGA logic size of accelerators, it is difficult to improve the inline compression ratio by enhancing dictionary compression. Therefore, there is a need for a technology that can improve the compression ratio of stored data while meeting the requirements of storage system I / O performance and the limitations on the circuit size for data compression. [Examples]

[0016] (1) System Configuration In the following, a storage system with a data compression function will be described as one embodiment of this specification. The storage system reduces the amount of stored data by lossless compression.

[0017] Figure 1 shows an example configuration of a storage system according to one embodiment of this specification. The storage system 101 includes a host interface 102, a storage controller 103, multiple solid state drives (SSDs) 105, and a cache memory 106 that utilizes volatile memory such as dynamic random access memory (DRAM).

[0018] The storage controller 103 is connected to the host interface 102, SSD 105, and cache memory 106, and includes a CPU 107, which is a processor that controls them, and memory 109. The CPU 107 may contain one or more cores and operates as a predetermined functional unit by executing programs (software) stored in memory 109.

[0019] Memory 109 stores system software, including the operating system necessary to run programs on the storage controller 103, programs that run on the processor 107, and metadata and data that programs use and need to be temporarily stored. Alternatively, only one of memory 109 or cache memory 106 may be implemented, and the data from the other may also be stored in memory 109.

[0020] The CPU 107 interprets the contents of read / write commands from the host (not shown), sends and receives data to and from the host, compresses and decompresses data using the LZMA compression / decompression circuit 104, which is a data compression / decompression circuit, and transfers data to and from the SSD 105 and cache memory 106. The CPU 107 also performs post-process compression of the data stored in the SSD 105.

[0021] The Host I / F102 is an interface mechanism for connecting to an external host, responding to read / write commands to send and receive data from the host. The mechanism of the Host I / F102 and the protocols for sending and receiving commands and data conform to, for example, standard interface specifications.

[0022] The storage controller 103 includes an LZMA compression / decompression circuit 104 and a transfer circuit 108. The transfer circuit 108 receives and transmits data that the LZMA compression / decompression circuit 104 compresses or decompresses. The transfer circuit 108 transfers data between components of the storage system 101, for example, between the LZMA compression / decompression circuit 104 and the cache memory 106, and between the CPU 107 and the cache memory 106. The transfer circuit 108 also transfers data between the host I / F 102 and the cache memory 106, and between the SSD 105 and the cache memory 106.

[0023] The LZMA compression / decompression circuit 104 reduces the amount of data stored in the SSD 105, which is a storage drive, by reversibly compressing the write data received in response to a write command and generating compressed data. In response to a read command, it decompresses the compressed data read from the SSD 105 to generate plaintext data in order to send the original plaintext data to the host.

[0024] The storage controller 103 performs inline compression and post-process compression. Inline compression compresses the data written from the host and stores it in the SSD 105. Post-process compression reads the compressed data from the SSD 105, decompresses the data, and returns it to the SSD 105.

[0025] In inline compression, the data written from the host is first temporarily stored in the cache memory 106. At this point, the storage controller 103 returns a write completion notification to the host. Subsequently, the data is converted into compressed data through the LZMA compression / decompression circuit 104, and the compressed data is also temporarily stored in the cache memory 106. Then, the compressed data is written to the SSD 105. As will be described later, the LZMA compression / decompression circuit 104 performs dictionary compression and range coding.

[0026] Post-process compression reads the compressed data from SSD 105 and stores it in cache memory 106. Next, post-process compression decompresses the compressed data back into plaintext using the LZMA compression / decompression circuit 104. Then, post-process compression uses the CPU 107 to compress the data again using dictionary compression, which has enhanced string search capabilities than the LZMA compression / decompression circuit 104, and range coding. The compressed data is temporarily stored in cache memory 106 and then written to SSD 105.

[0027] The data to be read from the host is read from the SSD 105 in a compressed state and temporarily stored in the cache memory 106. Then, it is converted to plaintext data through the LZMA compression / decompression circuit 104, and the plaintext data is also temporarily stored in the cache memory 106. Finally, the plaintext data is sent to the host.

[0028] The LZMA compression / decompression circuit 104 is implemented as hardware (logic circuit) designed based on a data compression / decompression method according to one embodiment of this specification, which differs from a software-operated processor. For example, it may be an accelerator using an FPGA. Because the LZMA compression / decompression circuit 104 has high-speed data decompression performance, the storage system 101 can take advantage of the high-speed random read performance that is a characteristic of SSDs, not only for uncompressed data but also for compressed data.

[0029] A different type of storage drive than an SSD, such as an HDD (Hard Disk Drive), may be used. The storage drive may not be implemented within the enclosure of the storage system 101, but may be connected to the storage controller 103 and cache memory 106 via a network, and the storage area of ​​the storage drive may reside in the cloud.

[0030] (2) LZMA algorithm As background information for explaining the data decompression method of one embodiment of this specification, the LZMA algorithm will be explained using Figures 2A to 5B.

[0031] (2-1) Overview of the LZMA algorithm Figure 2A shows an overview of the LZMA algorithm. In the LZMA compression process, the plaintext data 201 before compression is first subjected to dictionary compression 202. The dictionary compression result is then subjected to range coding 203. This generates the LZMA compressed data 204.

[0032] On the other hand, in the LZMA decompression process, the compressed data 204 is first subjected to range decoding 205. After that, the decoded result is subjected to plaintext decompression 206. This generates the original plaintext data 201.

[0033] (2-2) Dictionary compression process Figure 2B shows a concrete example of dictionary compression 202, which constitutes the LZMA algorithm. In the string stream of plaintext data 201, it is sequentially checked whether the same string appears again. If a string matches L consecutive characters starting from J characters before its first character, this string is converted into the copy symbol [L,J].

[0034] For example, the four-character string 211, "b,c,d,e", matches four consecutive characters starting from the sixth character "b". In this case, string 211 is converted to the copy symbol [4,6]. Similarly, the four-character string 212, "a,b,a,b", matches four consecutive characters starting from the second character "a" (including overlapping parts). In this case, string 212 is converted to the copy symbol [4,2].

[0035] Similarly, the six-character string "c,d,e,f,e,b" (string 213) matches the first character "c" starting from 15 characters prior to the first character, a sequence of 6 matching characters. In this case, string 213 is converted to the copy symbol [6,15]. Since the data size of these copy symbols is less than that of the original string, this conversion reduces the amount of data.

[0036] The range of the string stream referenced in the matching search (hereinafter referred to as the dictionary) is defined as the range from one character before to a predetermined number of characters before. Since the dictionary range slides backward with each search, this compression technique is also called sliding dictionary compression. If multiple matching strings exist within the dictionary range, the longest consecutive matching string is converted into a copy symbol. This has the effect of reducing the amount of data even further.

[0037] To generate the data to be input to the subsequent range encoding 203, it is necessary to encode the characters that were not converted to copy symbols (hereinafter referred to as literal characters) and the copy symbols using a predetermined bit pattern, and then concatenate them to form a bitstream.

[0038] Figure 2B shows the bitstream resulting from encoding according to the rules of the LZMA specification. This bitstream is input to range encoding 203. For example, bit pattern 221 is 12 bits long and represents the copy symbol [4,6]. Bit pattern 222 is 11 bits long and represents the copy symbol [4,2]. Bit pattern 223 is 13 bits long and represents the copy symbol [6,15]. Thus, the bit pattern length corresponding to the copy symbol is not fixed. On the other hand, literal characters are represented by a 9-bit bit pattern obtained by adding one zero to the beginning of the 8-bit value of the character.

[0039] The range decoding process 205 outputs such a bitstream during the LZMA decompression process. When the plaintext decompression process 206 receives such a bitstream as input, it interprets it as copy symbols and literal characters and restores the string stream of the plaintext data 201.

[0040] Figure 2C shows an example configuration of the hash table 230 used by the LZMA compression / decompression circuit 104 to search for matching strings in dictionary compression 202. The hash table 230 shows the 3-character string from which each hash value is obtained in the input data, and the position of that string. In the example shown in Figure 2C, each hash value is represented by 3 bits, and the string consists of 3 consecutive characters. Also, the upper limit of the number of entries for each hash value is 3.

[0041] The following describes the processing procedure for dictionary compression 202. The plaintext data, which is the input data, has the following structure as shown in Figure 2B: "a,b,c,d,e,f,e,b,c,d,e,a,b,a,b,a,b,c,d,e,f,e,b,…". The LZMA compression / decompression circuit 104 performs the following processing sequentially from the beginning of the input data.

[0042] [1] First, the LZMA compression / decompression circuit 104 calculates a 3-character hash value X (3 bits) from the current position and registers the current position and its 3 characters in the hash table 230. Note that, due to the characteristics of the hash function, the same hash value may be calculated from strings that do not match at all.

[0043] [2] Next, if the number of registered entries for hash value X exceeds 3, the LZMA compression / decompression circuit 104 discards the oldest entry (the entry with the smallest position value). If a string appears frequently, information is registered in the entry at a high frequency, so in the processing of step [3] below, entries with the same string will be hit with a high probability. On the other hand, strings that appear infrequently have a low priority from the standpoint of compression efficiency obtained by conversion to copy code, so the hash table can be used efficiently by preferentially removing such strings from the entries.

[0044] [3] Next, the LZMA compression / decompression circuit 104 refers to other entries related to the hash value X and checks for the existence of an entry that stores a string that matches the newly registered current string. If the strings stored in one or more entries for the hash value X are the same, it selects the string that is closest to the current string. This selection operation can be understood as follows, for example, assuming that "abc" is given as the current string again in Figure 2C. The new current string "abc" is registered in entry 3, and entry 2 which has a string that matches this current string is identified. On the other hand, if the hash value X is found but there is no string that matches the current string in the corresponding entry, the circuit proceeds to step [7].

[0045] [4] Next, since there is a possibility of four or more characters matching, the LZMA compression / decompression circuit 104 directly reads the input data and checks the longest match length L. That is, the current string has a matching string in the entry associated with the corresponding hash value X, and the range of matching characters following that string is also investigated. As a result, identical strings (three or more characters) that begin with the string stored in the entry are identified at two positions, and the length of these identified strings is calculated as the longest match length L.

[0046] [5] Next, the LZMA compression / decompression circuit 104 determines the current position (the position of the current string) and the distance D to the beginning of the found matching string (equal to the difference in the positions stored in the two compared entries).

[0047] [6] Next, the LZMA compression / decompression circuit 104 converts a string of length L from the current position into a copy code [L,D] and proceeds to step [8].

[0048] [7] If no string matching the current string exists in step [3], the LZMA compression / decompression circuit 104 converts the character at the current position into a literal code.

[0049] [8] After step [6], the LZMA compression / decompression circuit 104 terminates processing if it has reached the end of the data. Otherwise, the LZMA compression / decompression circuit 104 returns to step [3].

[0050] The dictionary code for the above plaintext data is as follows: "a,b,c,d,e,f,e,[4,6],a,b,[4,2],[6,15],…".

[0051] As described above, using hashtable 230 allows for efficient discovery of matching three-character strings. Then, by checking for matches of four or more characters, it's possible to find matching strings with more characters.

[0052] Generally, the larger the number of bits in the hash value (number of rows in the hash table 230), and the larger the upper limit on the number of entries (number of columns in the hash table 230), the more strings can be registered in the hash table 230, increasing exponentially by two, and thus improving the probability of finding a matching string. In other words, the compression ratio of the dictionary compression improves. However, improving the compression ratio requires a large amount of table memory, which increases the circuit size of the LZMA compression / decompression circuit 104. The probability of finding a matching string can also be improved by reducing the number of characters in the string. However, this increases the search time for matching strings in plaintext data. It is also possible to improve the probability of finding a matching string by increasing the number of entries allowed for a given hash value. That is, by increasing the period for which the string and its position are retained in an entry, even if its frequency of occurrence is low, the chances of finding a matching string are increased. However, adopting this method also increases the search time for matching strings.

[0053] (2-3) Range coding and decoding process Figure 3A shows the functional block diagram for range coding, and Figure 3B shows the functional block diagram for range decoding. Referring to Figure 3A, the range coding function 300 will be explained. The encoder 301 is an operation block that receives input bitwise from the input bit sequence 302 in 1-bit units and generates the output code 303. In the example in Figure 3A, the input bit sequence is "1,1,0,1", which corresponds to a part of the bit sequence shown in Figure 2B. The method for generating the output code 303 will be explained in the next section using Figures 4A and 4B. The output code 303 is compressed data using the LZMA algorithm (LZM in Figure 2A). A pressure Reduced data 204 This corresponds to ).

[0054] The encoder 301 also uses probability values ​​quoted from the probability table 304 as input. This probability value P(x) indicates the probability that the next input bit from the input bit sequence 302 is "0" given that the bit history 305 input up to that point is x. In this diagram, for the sake of simplicity, a common probability notation using decimals is used, but in implementation, a sufficiently large range of integers is given, and this range is divided into integer widths (partitions) corresponding to the probabilities. Furthermore, if the range of integers becomes too small to be suitable for further division, the number of digits is increased at that stage to allow for further division, thereby avoiding problems with floating-point numbers, etc. Alternatively, the probability value P(x) may also indicate the probability that the next input bit from the input bit sequence 302 is "1" given that the bit history 305 input up to that point is x.

[0055] The encoder 301 performs learning-based adaptation each time it uses the probability value P(x). For example, if the next input bit is actually "0", it increases P(x), and if it is "1", it decreases P(x). Note that at the start of encoding, when P(x) is unused, all values ​​are 0.5 (the probabilities of "0" and "1" are equal).

[0056] For example, in Figure 3A, if we consider the situation where the first bit is given, at this point x = "-". The probability that the first bit is "0" is 0.4, and the probability that it is "1" is 0.6. If the bit sequence is given immediately after the encoder is fully initialized, P(-) = 0.5 (initial value), but in Figure 3A, P "-" is different from the initial value as a result of bit history learning being applied.

[0057] Furthermore, when x=0, the probability that the second bit is "0" when the first bit is "0" is 0.35, and the probability that it is "1" is 0.65. In this case as well, if the second bit is "0", the probability of P(x=0) is increased, and if the second bit is "1", the probability of P(x=0) is decreased. These probability updates are learned throughout the entire bit history (bitstream) being studied. The compression mechanism by applying the probability table 304 obtained in this way to the bit history 305 is explained in Figure 4A.

[0058] Next, the range decoding function 310 will be described with reference to Figure 3B. The decoder 311 is an operation block that receives the input code 312 (i.e., the output code 303) and generates the output bit sequence 313 bit by bit. The method of generation will be explained in the next section using Figures 4A and 4B. The output bit sequence 313 corresponds to the input bitstream to the plaintext decompression process of the LZMA algorithm (plaintext decompression process 206 in Figure 2), that is, the output bitstream from the dictionary compression 202 in Figure 2.

[0059] Decoder 311 also uses probability values ​​taken from probability table 314 as input. This probability value P(x) indicates the probability that the next output bit is "0" given that the bit history output so far 315 is x.

[0060] Similar to the encoder 301, the decoder 311 performs learning-based adaptation each time the probability value P(x) is used. For example, if the next output bit is actually "0", P(x) is increased, and if it is "1", P(x) is decreased. Note that at the start of decoding, all unused values ​​of P(x) are 0.5 (the probabilities of "0" and "1" are equal).

[0061] When the output code 303 of the range coding function 300 and the input code 312 of the range decoding function 310 are the same, the changes in all probability values ​​P(x) in probability table 304 and probability table 314 due to learning will be the same. Therefore, the changes during coding are reproduced during decoding.

[0062] (2-4) Principles of Range Encoding and Decoding Figures 4A and 4B show examples illustrating the principle of range coding and decoding. The coding process performed by encoder 301 in Figure 3A divides the numerical axis [0, 1) into sections of length corresponding to the probability that each bit value in the input bit sequence is "0," and repeatedly leaves one of the sections as the target for division for the next bit. The coding process outputs the coordinate values ​​contained in the last remaining section as a code. Note that if the bit is "0," the left section of the division is left, and if the bit is "1," the right section of the division is left. The probability that each bit value is "0" is obtained from probability table 304, using the input bit history as an index.

[0063] According to the LZMA algorithm, the bit history used to reference probability values ​​is cleared under certain conditions. For example, in the case of a 9-bit literal character, the encoder 301 encodes the first bit of the eight bits (excluding the 1-bit header) using an empty (NULL) bit history. The encoder 301 encodes the last eighth bit using the first to seventh bits as the bit history, and then clears the bit history.

[0064] The decoding process performed by decoder 311 in Figure 3B divides the numerical axis [0, 1)] into segments of length corresponding to the probability that each bit value in the output bit sequence is "0," as shown in Figures 4A and 4B. Furthermore, decoder 311 checks which segment the input code (coordinate value) falls into, determines each bit in the output bit sequence, and repeatedly leaves the segment containing the code as the target for division for the next bit. Finally, it determines all the values ​​in the output bit sequence.

[0065] When the code is divided, if the code is included in the left-hand section, the decoder 311 determines that the bit is "0", and if the code is included in the right-hand section, the bit is determined to be "1". The probability that each bit value is "0" is obtained from the probability table 314, using the output bit history up to that point as an index.

[0066] Decoder 311 clears the bit history for referencing probability values ​​under the same conditions as encoding. For example, in the case of a 9-bit literal character, decoder 311 decodes using a bit history where the first bit is empty (NULL) among the 8 bits excluding the 1-bit header. Decoder 311 decodes the last 8th bit using bits 1-7 as the bit history, and then clears the bit history.

[0067] Figures 4A and 4B both show examples of numerical axis partitioning transitions in the encoding and decoding process of the bit sequence "1101". However, the probability values ​​for bit values ​​being "0" are different. In the example in Figure 4A, the probability value for "0" is always 0.5. In the example in Figure 4B, the probability values ​​for "0" are 0.25, 0.25, 0.75, and 0.25 bit by bit. Figure 4A shows the initial state when the probability values ​​referenced from probability tables 304 and 314 are used.

[0068] Therefore, if probability table 304 has already been learned, the intervals are divided according to the probabilities stored in the learned probability table 304. Also, when the bit string "1101" is entered, in Figure 4A, the "1" section in the 1st, 2nd, and 4th columns is made larger than the "0" section, and the "0" section in the 3rd column is made larger than the "1" section. The size of the "0" section in each row (the value that forms the boundary with "1" when the leftmost end of each row is considered to be 0) is then registered in probability table 304 (if the bit string "1101" is encoded, P(x) in the rows x="-", "1", "10", and "110" is updated). Figure 4B shows the case where the probability values ​​referenced from probability tables 304 and 314 have changed due to learning.

[0069] In range coding and decoding, if the bit sequence "1101" is processed frequently, probability tables 304 and 314 learn that "1" is more likely to appear first, if the history is "1", "1" is more likely to appear next, if the history is "11", "0" is more likely to appear next, and if the history is "110", "1" is more likely to appear next. As this probability learning progresses and the prediction of bit appearances becomes more accurate, the output code of the range coding becomes shorter.

[0070] The encoding process in Figure 4A is as follows: • Step 1: According to the input "1", the right 1 / 2 interval [1 / 2 to 2 / 2) remains. Step 2: The right half interval [3 / 4~4 / 4) remains according to the input "1". • Step 3: The left half interval [6 / 8~7 / 8) remains according to the input "0". • Step 4: The right half interval [13 / 16~14 / 16) remains according to the input "1". The last segment, 13 / 16 (1101 in binary), is set as output code 401.

[0071] On the other hand, the coding flow in Figure 4B is as follows: • Step 1: The right 3 / 4 interval (1 / 4 to 4 / 4) remains according to the input "1". Step 2: Following the input "1", the right 3 / 4 interval [7 / 16~16 / 16] remains. • Step 3: The left 3 / 4 interval (28 / 64~55 / 64) remains according to the input "0". • Step 4: The right 3 / 4 interval [139 / 256~220 / 256] remains according to the input "1". The 3 / 4 (11 in binary) included in the last section will be used as output code 411.

[0072] The more the bit sequence is input in line with the probability prediction (i.e., the more identical it is to the bit history processed in the past), the larger the size of the remaining interval after partitioning becomes. Therefore, fewer bits are needed to represent the coordinate values ​​of the output code included in the final remaining interval. In the example in Figure 4A, 4 bits are needed because the probability table is an initial value, whereas in the example in Figure 4B, 2 bits are sufficient as a result of applying a learned probability table. In this way, range coding improves its compression ratio by learning the bit occurrence probability according to the bit history.

[0073] The output codes 401 and 411 obtained as a result of the above encoding are given as input codes in the decoding process. For example, if output code 401 is given as the input code, first P(x) for the case x = "-" is obtained from the probability table 314. Then, for the probability of P(x = "-"), it is determined whether 13 / 16 is included in the "0" or "1" section. In the case of Figure 4A, since P(x) is set to 0.5 for all x values, P(x = "-") is calculated to be 1 / 2, and it is determined that 13 / 16 is in the range of 1 / 2 to 2 / 2 of the equally divided 0 / 2 to 2 / 2 section, and as a result, the first bit is identified as "1".

[0074] Next, since the first bit has been identified, probability table 314 is consulted to obtain P(x="1")=0.5, and it is determined whether 13 / 16 of output code 401 falls into the "0" or "1" section. In this case, since 13 / 16 falls within the 3 / 4~4 / 4 range in the 2 / 4~4 / 4 section, the second bit is identified as "1". Subsequently, the bit history (x=11) is used to consult the probability table again to find the section containing output code 401 and to identify the corresponding bit value. By repeating this operation sequentially, it becomes possible to decode the input bit sequence from the output code.

[0075] The probability for each bit history (x) in the probability table 314 is updated each time a bit is decoded, and the update follows the same process as when the encoder 301 generates the output code 303. That is, when the input code 312 is decoded from the beginning (i.e., identified from the first bit of the bit history 315), in the initial stage the output code 300 generated by the encoder 301 3 Similar to the initial generation process, interval partitioning is performed based on probability information close to the initial value, and the bit history 315 containing the input code 312 is identified. If the output code 303 and the input code 312 are the same, the bit history identified during interval partitioning will also be the same.

[0076] As decoding progresses, the probability table 314 is updated, and probability information corresponding to the characteristics of the bit history 315 appears. The interval partitioning corresponding to this probability information learned according to the characteristics of the bit history 315 is the same as the interval partitioning during the encoding process. Therefore, if the input code 312 and the output code 303 are the same, the bit sequence that is specified for the calculated interval partitioning is uniquely identified, and it becomes possible to decode the input bit sequence 302 as the output bit sequence 313.

[0077] (2-5) Flowchart for Range Encoding and Decoding Figure 5A shows a flowchart of an example of range coding. Figure 5B shows a flowchart of an example of range decoding. First, the procedure for range coding will be explained with reference to Figure 5A.

[0078] The encoder 301 references the probability table 304 for the probability that the next bit is "0" according to the input bit history (501). The encoder 301 divides the numerical axis range (the range to be divided) into two intervals according to that probability value (502). When dividing, it performs multiplication of the range size and the probability value. This is the most time-consuming part of the encoding process. Then, the encoder 301 selects one of the two intervals depending on whether the input bit value is "0" or "1" (503).

[0079] Next, in step 504, the encoder 301 determines if the bit input has ended. If the bit input has ended (504: YES), it proceeds to step 506; if there is still input (504: NO), it proceeds to step 505.

[0080] In step 505, the encoder 301 updates the probability values ​​used against the probability table 304 and updates the bit history for encoding the next bit. The probability values ​​are increased if the input bit value is "0" and decreased if it is "1". The bit history is updated, for example, if the next input bit after "11" is "0", it is changed to "110". After this, the encoder 301 returns to step 501 and continues the encoding process.

[0081] Meanwhile, in step 506, the encoder 301 outputs a coordinate value that identifies the last remaining section, for example, the value included in that section with the fewest number of represented bits, as the code, and completes the encoding process.

[0082] Referring to Figure 5B, the procedure for an example of range decoding will be explained. Decoder 311 refers to probability table 314 according to the output bit history and obtains the probability value that the next bit is "0" (511). Decoder 311 divides the numerical axis range into two intervals according to that probability value (512). When dividing, it performs multiplication of the range size and the probability value. This is the most time-consuming part of the decoding process. Then, decoder 311 selects the interval in which the value of the input code is contained (513). Decoder 311 outputs the bit value "0" or "1" that represents the selected interval (514).

[0083] Next, in step 515, the decoder 311 determines if the bit output has ended. If it has ended (515: YES), it finishes the decoding process. If there is still output (515: NO), it proceeds to step 516.

[0084] In step 516, the decoder 311 updates the probability value used against the probability table 314 and updates the bit history for decoding the next bit. The probability value is updated by increasing it if the output bit value is "0" and decreasing it if it is "1". vinegar. The bit history update involves, for example, changing the output bit following "11" to "110" if it is "0". After this, decoder 311 returns to step 511 and continues the decoding process.

[0085] Note that the above encoding involves both encoding and decoding of a given bit string ( The probability table is updated each time a bit of the symbol stream is processed, and the longer the bit sequence being processed, the more suitable the probability distribution for that bit sequence is obtained, increasing the compression efficiency (i.e., the further back in the bit sequence being processed you are, the more likely you are to get a higher compression effect).

[0086] In addition to this adaptive method of updating the probability table, it is also possible to create a probability table by checking the entire bit sequence before encoding begins, and then use this probability table as a fixed parameter for encoding and decoding. Compared to the adaptive method described above, when the probability table is created and used independently of the encoding and decoding process, the probability (interval division) is independent of the position within the bitstream being processed, making it possible to encode and decode from positions other than the beginning. However, in this case, the encoder 301 and decoder 311 must share the same probability table. (3) Method for speeding up range coding processing

[0087] Figure 6 shows a functional block diagram of an example of a high-speed range coding method. The LZMA compression / decompression circuit 104 in Figure 1 performs the compression processing shown in this block diagram. The example explained with reference to Figures 3A to 5B performs multiplication by referring to a probability each time a bit is input. Therefore, only one bit can be processed in one operation cycle, which can be a reason why the processing performance of the LZMA algorithm is slow.

[0088] The range coding function 600 shown in Figure 6 speeds up the range coding process by operating multiple encoders simultaneously. Specifically, N ranges to be divided are prepared (N>1), and N types of bit histories are prepared in advance from the input bit sequence. This range coding function 600 simultaneously references N probability values ​​in the probability table and performs multiplication operations on N input bits in parallel, thereby improving coding performance by an N-fold increase.

[0089] Figure 6 shows an example of encoding for N=4 using this acceleration technique. The four encoders 601A to 601D all perform the same processing as encoder 301 in Figure 3A. Each encoder retrieves and uses one probability value from the probability table 604. These four probability values ​​are referenced using the bit history 605A to 605D as an index.

[0090] Bit history 605A is used when the encoder 601A processes the first bit "1" of the input bit sequence 602, and its value is empty (NULL). Bit history 605B is used when the encoder 601B processes the second bit "1" of the input bit sequence 602, and its value is "1".

[0091] Bit history 605C is used when encoder 601C processes the third bit "0" of the input bit sequence 602, and its value is "11". Bit history 605D is used when encoder 601D processes the fourth bit "1" of the input bit sequence 602, and its value is "110".

[0092] In general terms, the bit history used to encode the Nth bit consists of a concatenation of the 1st to (N-1)th bits. By preparing four types of bit histories in this way, the four encoders 601A to 601D can simultaneously refer to four probability values ​​from the probability table 604 and perform multiplication simultaneously using those probability values.

[0093] The four subcodes 603A to 603D output from encoders 601A to 601D are concatenated at the end to form output code 606. Output code 606 corresponds to the compressed data of the LZMA algorithm. With this method, 4 bits of input can be processed in a single operation cycle, thus improving the range coding performance in the LZMA algorithm's compression process by four times compared to conventional methods.

[0094] Figure 7 shows an example flowchart of the high-speed range coding method described with reference to Figure 6. The procedure for the high-speed range coding method will be explained with reference to the flowchart in Figure 7. First, the LZMA compression / decompression circuit 104 creates N types of bit histories (701) which are used to encode N bits of the input bit sequence. N is an integer greater than or equal to 2. The N encoders obtain a probability value from a probability table that the next bit is "0" based on their respective bit histories (702). The N encoders divide each of the N numerical axis ranges (ranges to be divided) into two intervals according to these probability values ​​(703).

[0095] The multiplication of the range size and probability value by the N encoders is performed in parallel. In the first cycle, the numerical axis range (range size) is common to all N encoders, and is [0, 1) in the example shown in Figure 6. From the second cycle onward, the interval selected by the encoder in the previous cycle becomes the target numerical axis range (range size). Each encoder selects one of two intervals (left or right) depending on whether the input bit value is "0" or "1" (704).

[0096] Next, in step 705, the LZMA compression / decompression circuit 104 determines if the bit input has ended. If the bit input has ended (705: YES), it proceeds to step 707; if there is still input (705: NO), it proceeds to step 706. In step 706, the LZMA compression / decompression circuit 104 updates the N probability values ​​used in the probability table. The probability values ​​are updated by increasing them if the input bit value is "0" and decreasing them if it is "1".

[0097] After this, the LZMA compression / decompression circuit 104 returns to step 701 and continues the encoding process. For example, if N is 4 and an 8-bit literal character is to be encoded, the first cycle of this flow encodes the first 4 bits, and the second cycle encodes the last 4 bits. In the second cycle, the bit history used to encode the 5th bit is the bit sequence of the first 4 bits.

[0098] For example, if the input bit sequence is 6 bits, the first cycle may encode the first 4 bits or 3 bits, and the second cycle may encode the last 2 bits or 3 bits. The maximum value of the input bit sequence to the LZMA compression / decompression circuit 104 is 4, and bit sequences smaller than or equal to that can be encoded.

[0099] In step 707, each encoder generates a coordinate value that identifies the last remaining section, for example, the value containing the fewest number of bits that can be represented within that section. The LZMA compression / decompression circuit 104 outputs a bit sequence formed by concatenating these N values ​​as a code, and the encoding process is completed.

[0100] In the range coding example shown in Figure 6, the integer N > 1, the number of input bits is N bits, i.e., the length of the bit history is a maximum of (N-1) bits. This range coding can speed up coding by N times by using N ranges to divide the data. As mentioned in the explanation referring to Figure 7, when the integer M > N and the number of input bits is M bits, i.e., the length of the bit history is a maximum of (M-1) bits, the coding of the input bits can be sped up by using N ranges to divide the data.

[0101] The method will be explained below using the case where M=8 and N=4 as an example. The LZMA compression / decompression circuit 104 has a probability table with 255 entries, indexed by a bit history of up to 7 bits. The LZMA compression / decompression circuit 104 prepares four types of bit histories (empty, 1 bit, 2 bits, and 3 bits, respectively) to be used for encoding the first 4 bits of an 8-bit input bit sequence, and simultaneously references the four corresponding probability values ​​in the probability table. The LZMA compression / decompression circuit 104 uses these probability values ​​to encode the first 4 bits in parallel in the first cycle.

[0102] Next, the LZMA compression / decompression circuit 104 prepares four types of bit histories (4 bits, 5 bits, 6 bits, and 7 bits, each including the first four bits) to be used for encoding the latter four bits of the input bits, and simultaneously references four corresponding probability values ​​from the probability table. Using these probability values, the LZMA compression / decompression circuit 104 encodes the latter four bits in parallel in the second cycle.

[0103] In this way, the LZMA compression / decompression circuit 104 processes the 8-bit input in 2 cycles (i.e., 4 times the performance), generates four subcodes, and constructs the output code by concatenating them.

[0104] In general, the encoding process for a range code with M bits as input can be improved by using a probability table of (2^M-1) entries indexed by the bit history of up to (M-1) bits and N encoders, and processing the M bits in [M / N] operation cycles. The LZMA compression / decompression circuit 104 may generate subcodes without performing the above parallel processing.

[0105] (4) Method for speeding up range decoding An N-fold speedup in range decoding cannot be achieved simply by running N decoders 311 in parallel as shown in Figure 3B. This is because the bit history used in the processing of a given decoder X remains uncertain until decoder Y, which decodes the previous bit, outputs its processing result. Therefore, decoder X cannot simultaneously reference probability values ​​from the probability table and perform multiplication using those values, as decoder Y does not, and thus parallelization cannot be achieved.

[0106] In the following, a method for accelerating range decoding processing according to one embodiment of this specification will be described. Figure 8 shows a functional block diagram of the method for accelerating range decoding processing. The LZMA compression / decompression circuit 104 in Figure 1 performs decompression processing by the range decoding function 800 shown in this block diagram.

[0107] Figure 8 shows an example of range decoding when N=4, i.e., when the number of bits in the output bit sequence is 4. The 15 decoders 8A (1), 8B0 and 8B1 (2), 8C00~8C11 (4), and 8D000~8D111 (8) all process in the same way as decoder 311 in Figure 3B. In Figure 8, the illustration of some decoders is omitted. The four subcodes 803A~803D input to these 15 decoders are separated from input code 802 (corresponding to the compressed data of the LZMA algorithm) and are the same as the four subcodes 603A~603D in Figure 6. Note that one subcode may be shared by multiple decoders.

[0108] Each decoder retrieves one probability value from probability table 804 and uses it to output candidate bit values. These 15 probability values ​​are referenced using all possible bit histories as an index.

[0109] The bit history value used by one decoder 8A to decode the first bit of the output bit sequence 806 is empty (NULL). The bit history values ​​used by two decoders 8B0 and 8B1 to decode the second bit of the output bit sequence 806 are "0" and "1", respectively.

[0110] The bit history values ​​used by the four decoders 8C00 to 8C11 to decode the third bit of the output bit string 806 are "00", "01", "10", and "11", respectively. The bit history values ​​used by the eight decoders 8D000 to 8D111 to decode the fourth bit of the output bit string 806 are "000", "001", "010", "011", "100", "101", "110", and "111", respectively.

[0111] In general terms, the number of bit histories used to decode the K-th bit is 2^(K-1). Each bit history is a (K-1) bit pattern (bit sequence) that is possible as the 1st to (K-1)th bits of the output bit sequence 806. By preparing 15 types of bit histories in this way, 15 probability values ​​are simultaneously referenced from the probability table 804, and the 15 decoders simultaneously perform multiplication using those probability values.

[0112] If the first bit of the output bit sequence 806 output by decoder 8A is "1", then the second bit output by decoder 8B1, which decoded assuming the first bit is "1", is the correct result. Therefore, selector 805B selects the "1" output by decoder 8B1 from the two candidates for the second bit output by decoders 8B0 and 8B1. In other words, the first and second bits are determined to be "11".

[0113] This confirms that the third bit output by decoder 8C11, which decoded assuming that the first and second bits of decoders 8C00 to 8C11 are "11", is the correct result. Therefore, selector 805C selects the "0" output by decoder 8C11 from the four candidates for the third bit output by decoders 8C00 to 8C11. In other words, the first to third bits are confirmed to be "110".

[0114] This confirms that the fourth bit output by decoder 8D110, which decodes assuming that the first to third bits of decoders 8D000 to 8D111 are "110", is the correct result. Therefore, selector 805D selects the "1" output by decoder 8D110 from the eight candidates for the fourth bit output by decoders 8D000 to 8D111.

[0115] Based on the above, it is confirmed that the 4 bits of the output bit sequence 806 are "1101". In general, the LZMA compression / decompression circuit 104 has 2^(K-1) decoders to decode the Kth bit and holds 2^(K-1) candidate Kth bits that they output. The LZMA compression / decompression circuit 104 selects as the Kth bit a candidate output by one decoder that has been decoded assuming that the values ​​of the already determined 1st to (K-1) bits are bit history.

[0116] The bit selection process performed by selectors 805B to 805D takes significantly less time than the decoder's multiplication process. Therefore, this method allows for the processing of 4 bits of output in a single operation cycle. As a result, the range decoding performance in the LZMA algorithm's expansion process is improved to four times that of conventional methods.

[0117] Referring to Figure 9, the procedure for the high-speed range decoding method described in Figure 8 is shown below. First, the LZMA compression / decompression circuit 104 creates (2^N-1) bit histories that may be used to decode N bits of the output bit sequence (901). The bit histories used to decode the Kth bit are 2^(K-1).

[0118] Each of the (2^N-1) decoders obtains a probability value from the probability table 804 that the next bit is "0" according to the bit history it is responsible for (902), and divides the numerical axis range (the range to be divided) into two intervals according to that probability value (903). The numerical axis range divided by the 2^(K-1) decoders used to decode the K bit is common. Specifically, in the first cycle, the numerical axis range of all decoders is common, which is [0,1) in the example in Figure 8. From the second cycle onward, the numerical axis range of the decoder for the K bit is the range of the division result by the decoder that output the correct answer for the K bit in the previous cycle. When dividing, the range size is multiplied by the probability value.

[0119] The decoder selects one of two intervals in which the input subcode value is contained (904), and generates a bit value of "0" or "1" that represents the selected interval (905). The number of bit values ​​generated is (2^N-1), and the number of candidates for the Kth bit is 2^(K-1). The selector then selects one correct bit from each candidate in order from the first bit, determines the N-bit pattern, and outputs it (906). The correct values ​​of the first to (K-1)th bits are used as the bit history to select the correct value for the Kth bit.

[0120] Next, step 90 7 In this case, the LZMA compression / decompression circuit 104 determines if the bit output is finished, and if it is finished, (90 7 :YES), after the decryption process is complete, if there is still output (90 7 :NO), Step 90 8 Move on.

[0121] Step 90 8Next, the LZMA compression / decompression circuit 104 updates the N probability values ​​used for the probability table 804. The probability values ​​are updated by increasing them if the output bit value is "0" and decreasing them if it is "1". Furthermore, the LZMA compression / decompression circuit 104 adopts the interval selected in step 904 by the decoder that output the correct bit value as the numerical axis range for the next cycle. The interval selected in step 904 by the one decoder that output the correct value for the K bit out of 2^(K-1) decoders for the K bit is adopted as the numerical axis range to be divided in step 903 during the decoding of the next K bit.

[0122] After this, the LZMA compression / decompression circuit 104 returns to step 901 and continues the decoding process. For example, if the 8 bits of a literal character are encoded in two cycles, with the first 4 bits and the last 4 bits being encoded, the bit history used to decode the 5th bit in the second cycle of this flow is the bit sequence of the first 4 bits.

[0123] For example, if 6-bit input data is encoded in two cycles, divided into the first 4 bits and the remaining 2 bits, the LZMA compression / decompression circuit 104 may decode 4 bits or 3 bits in the first cycle, and then decode 2 bits or 3 bits in the second cycle. The maximum value of the input bit sequence to the LZMA compression / decompression circuit 104 is 4, and it can decode bit sequences smaller than or equal to that.

[0124] In the acceleration method for decoding range codes shown in Figure 8, when the integer N > 1, the number of input bits is N bits (i.e., the length of the bit history is a maximum of (N-1) bits), and N ranges are used to accelerate the process by N times. As mentioned in Figure 9 with the example of an 8-bit literal character, when the integer M > N and the number of input bits is M bits, i.e., the length of the bit history is a maximum of (M-1) bits, the decoding of the output bit sequence can be accelerated by using N ranges to be divided.

[0125] The following describes an example of decoding an 8-bit output bit sequence. The LZMA compression / decompression circuit 104 has 15 decoders, similar to Figure 8, and four subcodes separated from the input code (corresponding to the compressed data of the LZMA algorithm) are input to these 15 decoders, similar to Figure 8. Each decoder obtains and uses one probability value from a 255-entry probability table, which uses a maximum of 7 bits of bit history as an index.

[0126] The 15 probability values ​​referenced in the first cycle are values ​​referenced using all possible bit histories (empty, 1 bit, 2 bits, and 3 bits, respectively) as indices for the first 4 bits of the 8-bit output bit string. The bit histology value used by the single decoder that decodes the first bit of the output bit string is empty (NULL).

[0127] The bit history values ​​used by the two decoders that decode the second bit of the output bit string are "0" and "1", respectively. The bit history values ​​used by the four decoders that decode the third bit of the output bit string are "00", "01", "10", and "11", respectively. The bit history values ​​used by the eight decoders that decode the fourth bit of the output bit string are "000", "001", "010", "011", "100", "101", "110", and "111", respectively.

[0128] The 15 decoders perform multiplication in parallel using the probability values ​​referenced in their bit history. Then, as in Figure 8, the selector's bit selection process sequentially determines the values ​​of the first to fourth bits. Here, for example, let's say "1101".

[0129] Next, the 15 probability values ​​referenced in the second cycle are values ​​that reference all possible bit histories (4 bits, 5 bits, 6 bits, and 7 bits, respectively, each containing "1101" which was determined in the first cycle) as the last 4 bits of the 8-bit output bit sequence.

[0130] The bit history value used by one decoder to decode the 5th bit of the output bit string is "1101". The bit history values ​​used by two decoders to decode the 6th bit of the output bit string are "11010" and "11011", respectively. The bit history values ​​used by four decoders to decode the 7th bit of the output bit string are "110100", "110101", "110110", and "110111", respectively.

[0131] The bit history values ​​used by the eight decoders that decode the 8th bit of the output bit string are "1101000", "1101001", "1101010", "1101011", "1101100", "1101101", "1101110", and "1101111", respectively.

[0132] The 15 decoders perform multiplication in parallel using the probability values ​​referenced in their bit history. Then, as in Figure 8, the selector's bit selection process sequentially determines the values ​​from the 5th bit to the 8th bit.

[0133] Similar to Figure 8, the bit selection process of the selector takes significantly less time than the multiplication process of the decoder. Therefore, this method allows for the processing of an 8-bit output in two operation cycles. As described above, in a probability table of 255 entries, 15 entries are referenced in the first cycle, and 15 entries are selected and referenced from the remaining 240 entries in the second cycle. In the second cycle, the number of entries referenced is narrowed down to 1 / 16 by indexing with the bit history, which includes the first 4 bits determined in the first cycle.

[0134] In general, the decoding process for a range code that outputs M bits can be improved by using a probability table of (2^M-1) entries indexed by the bit history of up to (M-1) bits and (2^N-1) decoders, processing the M bits in [M / N] operation cycles.

[0135] As described above, one embodiment of this specification can decompress data compressed by range coding at high speed. Therefore, for example, in a device storage system having a data compression function using a range coding algorithm, the read response performance of the compressed data can be improved.

[0136] (5) Post-process compression Post-process compression is described below. Post-process compression can improve the compression ratio of data stored in SSD105 while suppressing the impact on write access from the host to the storage system. Figure 10 is a logical configuration diagram illustrating post-process compression, and Figure 11 is a flowchart illustrating post-process compression.

[0137] Referring to Figure 10, inline compression compresses the write data 211 from the host using the LZMA compression / decompression circuit 104 and stores the inline compressed data in the SSD 105. Post-process compression decompresses the inline compressed data stored in the SSD 105 using the LZMA compression / decompression circuit 104, recompresses the decompressed data using the CPU 107, and stores it in the SSD 105.

[0138] The CPU 107 performs range coding 172 after dictionary compression 171, which is compatible with range coding 203 in the LZMA compression / decompression circuit 104 (using the same compression / decompression algorithm). The dictionary compression 171 (second dictionary compression) by the CPU 107 has a higher search capability for matching strings than the dictionary compression 202 (first dictionary compression) by the LZMA compression / decompression circuit 104. This makes it possible to increase the compression ratio of post-process compression above that of inline compression. Furthermore, since the range coding 172 of the CPU 107 is compatible with range coding 203 in the LZMA compression / decompression circuit 104, decompression can be performed using the LZMA compression / decompression circuit 104 during read processing, thereby suppressing a decrease in read performance.

[0139] Range coding 203 and 172 may be omitted, and coding using other compression / decompression algorithms may be performed. Range coding may be performed using a different process than the parallel processing described above. Data compression ratio can be increased by performing dictionary compression and coding different from dictionary compression, such as entropy coding like range coding or Huffman coding.

[0140] Referring to Figure 11, in step 1001, CPU 107 compares its current utilization rate with a preset threshold to determine if the utilization rate is below the threshold. If the utilization rate is above the threshold (1001: NO), this flow terminates. If the utilization rate is below the threshold (1001: YES), this flow proceeds to step 1002.

[0141] By initiating post-process compression when the CPU load of CPU107 is below a threshold, the impact on other processes can be minimized. A value representing CPU load different from CPU utilization, such as the number of running tasks, may be referenced. The CPU load does not necessarily have to be referenced; for example, post-process compression may be performed periodically.

[0142] For example, CPU 107 may perform post-process compression along with garbage collection. CPU 107 adds updated data at a certain address in the volume to a new address in SSD 105. The old data at the above address stored in SSD 105 becomes invalid data. Garbage collection stores the valid data in SSD 105 together in a new address area and changes the area of ​​invalid data to free space. When moving valid data, CPU 107 performs post-process compression on the target data read from SSD 105 and stores it at the new address in SSD 105. This allows for efficient post-process compression.

[0143] In step 1002, the CPU 107 selects and reads one of the inline compressed data from the SSD 105 and stores it in the cache memory 106. As described above, inline compression compresses the data written from the host by the LZMA compression / decompression circuit 104 and stores it in the SSD 105.

[0144] Data stored in SSD105 that has not undergone post-processing compression may be managed by management information (not shown). For example, the management information may include addresses within the volume, information on which post-processing compression has not been completed, and address information of SDD105 that stores the valid data for those addresses. The management information may be stored in the memory of the storage controller 103.

[0145] The management information may include information about whether or not the valid stored data has undergone post-processing compression, as well as information about the time the data was stored (updated). The CPU 107 may select data to perform post-processing compression on based on the data's update time. For example, among the candidate data for post-processing compression, the CPU 107 may prioritize data with older update times. Candidate data are valid data that has not yet undergone post-processing compression. The CPU 107 may select from the oldest data, or it may select data to perform post-processing compression on from data whose elapsed time since update exceeds a threshold.

[0146] In step 1003, the LZMA compression / decompression circuit 104 decompresses the inline compressed data stored in the cache memory 106 in response to instructions from the CPU 107 and stores it back in the cache memory 106. By using the LZMA compression / decompression circuit 104, data can be decompressed efficiently in a short amount of time.

[0147] In step 1004, CPU 107 performs dictionary compression 171, which enhances string search capabilities compared to inline compression. This enables a higher compression ratio than inline compression. For example, CPU 107 performs dictionary compression using a hash table with a large number of hash bits or a large upper limit on the number of entries. Alternatively, CPU 107 may perform dictionary compression using a hash table with a small number of characters per string.

[0148] The hash table may have both a hash bit count and an upper limit on the number of entries greater than the values ​​obtained with inline compression. Furthermore, the number of hash bits and / or the upper limit on the number of entries may be greater than the values ​​obtained with inline compression, and the number of characters in a string may be smaller than the values ​​obtained with inline compression. The hash table may be stored in the memory of the storage controller 103.

[0149] In step 1005, the CPU 107 performs range coding 172, which is compatible with the range coding 203 of the LZMA compression / decompression circuit 104, and encodes the data into multiple subcodes. This enables high-speed processing using the LZMA compression / decompression circuit 104 when reading compressed data.

[0150] In step 1006, the CPU 107 stores the output code (post-process compressed data (post-compressed data in Figure 11)) in the cache memory 106. In step 1007, the CPU 107 writes the post-process compressed data to an address region different from the original inline compressed data in the SSD 105. In step 1008, the selected inline compressed data is disabled and the post-process compressed data is enabled. Specifically, the CPU 107 updates the management information that manages the address of the SSD 105 and the enabled / disabled status of the stored data.

[0151] Alternatively, in step S1006, the post-processed compressed data may be stored in the cache memory, and then, in parallel with the operation of writing the post-processed compressed data from the cache memory to the SSD 105, the data may be transferred and stored in a cloud storage area to perform a backup.

[0152] In addition, by configuring the system to restrict backups to the cloud when writing inline compressed data to the SSD105, it is possible to achieve enhanced data preservation performance while reducing cloud capacity utilization in a hybrid environment combining on-premises and cloud systems. Data transfer to the cloud does not have to occur at times other than those mentioned above; the entire system can be built as a hybrid cloud system to periodically transfer post-processed compressed data.

[0153] Furthermore, regarding the backup function that utilizes cloud storage capacity, it is possible to apply processes from S1006 onwards (storage of post-processed compressed data) only to data storage on the cloud. That is, in S1006, post-processed compressed data is stored in cache memory, and then that post-processed compressed data is transferred to the storage area on the cloud.

[0154] In this process, the selection of data to which post-processing compression should be applied can be based on, for example, the recent usage frequency of the inline-compressed data. For instance, by monitoring the aforementioned usage frequency, data whose usage frequency falls below a predetermined frequency (i.e., low usage frequency) can be identified as data to which post-processing compression should be applied. Such data can then be transferred to the cloud after post-processing compression, and the corresponding inline-compressed data can be invalidated according to the S1008 process.

[0155] With such a hybrid system, frequently accessed data can be placed in the on-premises environment to ensure responsiveness, while less frequently accessed data can be placed in the cloud environment with high compression ratios. In other words, it becomes possible to achieve both responsiveness in data reading and writing and efficient use of storage space. [Examples]

[0156] Other embodiments of post-process compression are described below. The differences from Embodiment 1 are mainly explained below. The description of Embodiment 1 can be applied to configurations similar to those in Embodiment 1. Figure 12 is a logical configuration diagram illustrating other embodiments of post-process compression, and Figure 13 is a flowchart illustrating other embodiments of post-process compression.

[0157] Referring to Figure 12, the post-process compression in this embodiment compresses the data dictionary-compressed by the CPU 107 using the range coding 203 of the LZMA compression / decompression circuit 1104. This reduces the load on the CPU 107 and speeds up the compression process. The LZMA compression / decompression circuit 1104 includes a selector 209 in addition to the configuration of the LZMA compression / decompression circuit 104 in Embodiment 1. The selector 209 selects the dictionary compression 202 in inline compression and the bit sequence 215 of the dictionary compression result by the CPU 107 in post-process compression, and outputs it to the range coding 203.

[0158] Referring to Figure 13, steps 1201 to 1204 are the same as steps 1001 to 1004 in the flowchart of Figure 11. In step 1205, the CPU 107 compresses the dictionary-compressed data by the range coding 203 of the LZMA compression / decompression circuit 1104. Steps 1206 to 1208 are the same as steps 1006 to 1008 in the flowchart of Figure 11.

[0159] Alternatively, in step S1206, the post-processed compressed data may be stored in the cache memory, and then, in parallel with the operation of writing the post-processed compressed data from the cache memory to the SSD 105, the data may be transferred and stored in a cloud storage area to perform a backup.

[0160] In addition, by configuring the system to restrict backups to the cloud when writing inline compressed data to the SSD105, it is possible to achieve enhanced data preservation performance while reducing cloud capacity utilization in a hybrid environment combining on-premises and cloud systems. Data transfer to the cloud does not have to occur at times other than those mentioned above; the entire system can be built as a hybrid cloud system to periodically transfer post-processed compressed data.

[0161] Furthermore, regarding the backup function that utilizes cloud storage capacity, it is possible to apply, for example, the process from S1206 onwards (storage of post-processed compressed data) only to data storage on the cloud. That is, in S1206, post-processed compressed data is stored in cache memory, and then that post-processed compressed data is transferred to the storage area on the cloud.

[0162] In this process, it is also possible to select data to which post-processing compression should be applied based on factors such as the recent frequency of use of inline-compressed data. For example, by monitoring the aforementioned frequency of use, data whose frequency of use falls below a predetermined frequency (i.e., low frequency of use) can be identified as data to which post-processing compression should be applied. Such data can then be transferred to the cloud after post-processing compression, and the corresponding inline-compressed data can be invalidated according to the S1208 process.

[0163] With such a hybrid system, frequently accessed data can be placed in the on-premises environment to ensure responsiveness, while less frequently accessed data can be placed in the cloud environment with high compression ratios. In other words, it becomes possible to achieve both responsiveness in data reading and writing and efficient use of storage space.

[0164] Furthermore, the storage systems in Examples 1 and 2, and their modified versions, can conserve resources by reducing the number of storage drives used through a reduction in the amount of data, thereby reducing the storage capacity used and the power consumption of the storage drives.

[0165] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail for the purpose of explaining the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0166] Furthermore, some or all of the above configurations, functions, and processing units may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, the above configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card or SD card.

[0167] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In practice, it is reasonable to assume that almost all components are interconnected. [Explanation of symbols]

[0168] 101…Storage system, 103…Storage controller, 104, 1104…LZMA compression / decompression circuit, 107 CPU, 171 Dictionary compression, 172 Range coding, 301, 601A~D…Encoder, 311, 8A, 8B0, 8B1, 8C00, 8C11, 8D000, 8D111…Decoder, 304, 314, 604, 804…Probability table, 302, 602…Input bit sequence, 303, 606…Output code, 312, 802…Input code, 313, 806…Output bit sequence

Claims

1. It is a storage system, The system includes a processor and a controller which includes a data compression / decompression circuit. The controller performs inline compression of plaintext data from the host and post-process compression of the inline compressed data stored in one or more storage drives. The aforementioned inline compression is The data compression / decompression circuit performs a compression process, including first dictionary compression, on the plaintext data from the host to generate inline compressed data. This includes storing the inline compressed data in one or more storage drives. The aforementioned post-process compression is The inline compressed data read from the one or more storage drives is decompressed by the data compression / decompression circuit to generate the plaintext data. The process includes performing a compression operation on the plaintext data, including a second dictionary compression performed by the processor having superior string search capabilities compared to the first dictionary compression, to generate post-process compressed data, and storing it in one or more storage drives. A storage system in which at least one of the hash bit count and entry limit of the hash table of the second dictionary compression is greater than that of the hash table of the first dictionary compression, and / or the number of characters in the string of the hash table of the second dictionary compression is less than that of the hash table of the first dictionary compression.

2. A storage system according to claim 1, The compression process by the data compression / decompression circuit includes, after performing the first dictionary compression, encoding using a first compression / decompression algorithm different from the first dictionary compression. The post-process compression is a storage system in which the processor compresses the plaintext data using the second dictionary compression, and then performs encoding using the first compression / decompression algorithm to generate the post-process compressed data.

3. A storage system according to claim 2, The post-process compression is performed by the processor in a storage system that performs encoding using the first compression / decompression algorithm.

4. A storage system according to claim 2, The post-process compression is a storage system in which the data compressed by the second dictionary compression is input to the data compression / decompression circuit and encoding is performed by the first compression / decompression algorithm.

5. A storage system according to claim 1, The controller is a storage system that starts post-process compression when the utilization rate of the processor is less than a preset threshold.

6. A storage system according to claim 1, The post-process compression is a storage system that selects the oldest data from among the candidate data stored in one or more storage drives.

7. The storage system according to Claim 1, The controller is a storage system that performs the post-process compression during garbage collection of the one or more storage drives.

8. The storage system according to claim 2, The controller is a storage system that reads the compressed data obtained by post-process compression from one or more storage drives, decompresses it into plain text data using the data compression / decompression circuit, and transmits it to the host.

9. A data compression method in a storage system, The aforementioned storage system, It includes a processor and a data compression / decompression circuit, The aforementioned data compression method is This includes inline compression of plaintext data from the host and post-process compression of inline compressed data stored on one or more storage drives. The aforementioned inline compression is The data compression / decompression circuit performs a compression process, including first dictionary compression, on the plaintext data from the host to generate inline compressed data. This includes storing the inline compressed data in one or more storage drives. The aforementioned post-process compression is The inline compressed data read from the one or more storage drives is decompressed by the data compression / decompression circuit to generate the plaintext data. The process includes performing a compression operation on the plaintext data, including a second dictionary compression performed by the processor having superior string search capabilities compared to the first dictionary compression, to generate post-process compressed data, and storing it in one or more storage drives. A data compression method wherein at least one of the hash bit count and entry limit of the hash table of the second dictionary compression is greater than that of the hash table of the first dictionary compression, and / or the number of characters in the string of the hash table of the second dictionary compression is less than that of the hash table of the first dictionary compression.

Images