Semi-classified compression using encoding and decoding tables

Abstract

A data processing platform, method, and program product perform compression and decompression on a set of data items. Suffix data and prefixes are selected for each corresponding data item in the set based on the data content of the corresponding data item. The set of data items are classified based on the prefixes. The prefixes are encoded by querying multiple encoding tables to create codewords containing compressed information representing the values ​​of all prefixes in the set of data items. The codewords and suffix data for each of the data items are stored in a memory. The codewords are decompressed to recover the prefixes. The recovered prefixes are paired with their corresponding suffix data.

Description

Background Art

[0001] Semi-classification has been described as being used to compress fingerprints in cuckoo filters and d-left counting Bloom filters, two important approximate set membership data structures (ASMDS) ubiquitous in networking hardware and software, as well as in other applications (e.g., database systems, genome sequencing, and file systems). A fingerprint is a short hash that encodes the presence of an item in the set approximately represented by the ASMDS. Typically, fingerprints are stored in buckets within the ASMDS (similar to cache sets), where the location of the fingerprint within the bucket does not change its meaning.

[0002] The prior art sorts these fingerprints by their prefixes and replaces the prefixes with codewords. To encode and decode the fingerprints, the prior art uses a pair of encoding and decoding tables. However, the growth rate of these tables relative to the associativity effectively limits the associativity of the buckets to 4 or less, as the lookup table size can be several kilobytes at an associativity of 4. Therefore, a solution that reduces the size of these tables to improve their storage in hardware cache or other memory would be ideal. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A data processing platform including compression according to some embodiments is illustrated in block diagram form.

[0004] Figure 2 A process for compressing and decompressing data according to some embodiments is illustrated in flow chart form.

[0005] Figure 3 A more detailed process for creating codewords containing compressed data according to some embodiments is illustrated in flow chart form.

[0006] Figure 4 The process for creating codewords containing compressed data using an encoding table according to some embodiments is illustrated in table form.

[0007] Figure 5 is a diagram illustrating a semi-classification process according to some embodiments.

[0008] Figure 6 is a diagram showing an example of using a coding table to create a codeword according to some embodiments.

[0009] Figure 7 A process for decompressing a codeword according to some embodiments is illustrated in flow chart form.

[0010] Figure 8A portion of a process for decompressing a codeword using an encoding table according to some embodiments is illustrated in diagram form.

[0011] Figure 9 Illustrated in diagram form Figure 8 Another part of the process.

[0012] Figure 10 An example of compression performance achieved according to some exemplary embodiments is illustrated in graphical form.

[0013] In the following description, the same reference numerals are used in different drawings to indicate similar or identical items. Unless otherwise stated, the term "coupled" and its associated verb forms include direct connections and indirect electrical connections by methods known in the art, and unless otherwise stated, any description of a direct connection implies the use of an alternative embodiment suitable for an indirect electrical connection form. DETAILED DESCRIPTION

[0014] A data processing platform includes a memory and a processor coupled to the memory, the processor capable of performing compression and decompression on a set of data items. The processor receives the set of data items and selects suffix data and prefixes for each corresponding data item in the set based on the data content of the corresponding data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying multiple encoding tables to create codewords containing compressed information representing the values ​​of all prefixes of the set of data items. The suffix data and codewords for each of the data items are stored in the memory. The codewords are decompressed to recover the prefixes, and the recovered prefixes are paired with their corresponding suffix data.

[0015] A method compresses and stores data items. A set of data items is received, and suffix data and prefixes are selected for each corresponding data item in the set based on the data content of the corresponding data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying multiple encoding tables to create code words containing compressed information representing the values ​​of all prefixes in the set of data items. The code words and suffix data for each of the data items are stored. The code words are decompressed to recover the prefixes. The recovered prefixes are paired with their corresponding suffix data.

[0016] A tangible, non-transitory, computer-readable medium stores a program product executable by at least one processor for compressing and decompressing data items. A set of data items is received, and suffix data and prefixes are selected for each corresponding data item in the set based on the data content of the corresponding data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying a plurality of encoding tables to create a codeword containing compressed information representing the values ​​of all prefixes in the set of data items. The codeword and suffix data for each of the data items are stored in a memory. The codeword is decompressed to recover the prefixes. The recovered prefixes are paired with their corresponding suffix data.

[0017] Figure 1 A data processing platform including compression according to some embodiments is illustrated in block diagram form. Data processing platform 100 can be any platform that needs to compress a set of data items with high entropy that can be reordered. For example, the techniques herein have many applications, such as compression of software and hardware cache tags and fingerprints in approximate set membership data structures. Generally speaking, data processing platform 100 provides compression and decompression for a set of fixed-length items. The techniques are most beneficial for uniformly random items (i.e., with maximum information entropy) whose order can be permuted. Applications may vary due to networking, software systems, and hardware caches.

[0018] Data processing platform 100 communicates with a network or host system 10 to fulfill requests for data storage and retrieval, or to perform operations for the host system, such as encryption key retrieval or caching. In this example, data processing platform 100 includes a processor 110 in communication with random access memory (RAM) 112, which may be internal or external to processor 110. For example, RAM 112 may be part of an integrated cache memory of processor 110 or in the main memory of platform 100. Processor 110 also communicates with non-volatile memory 120, which provides a tangible, non-transitory computer-readable medium, such as flash memory or a hard drive, that stores computer program code, such as compression and decompression program code 122.

[0019] RAM 112 stores data used in the compression techniques described below, such as codeword σ and suffixes 0 through k-1 (there are k suffixes, each associated with one of the k data items). Such data may also be stored in non-volatile memory. When program code 122 is loaded from non-volatile memory 120, the compression and decompression program code is present in RAM 112, as depicted by the loaded compression engine 114 and the loaded decompression engine 116. Other data used with the techniques described herein, such as data items to be compressed and encoding tables, may also be stored in RAM 112 or non-volatile memory 120.

[0020] Although in this embodiment the data processing platform includes a processor 110 that executes program code instructions, other embodiments may implement some or all compression features in hardware, such as digital logic in a networked ASIC, a CPU, a GPU, or programmable logic in an FPGA.

[0021] Figure 2 A process 200 for compressing and decompressing data according to some embodiments is illustrated in flowchart form. At block 202, a set of data items is provided for compression. The data items are typically fixed-length items, typically having high entropy as discussed above. Some subsequences of bits in the data items may have high entropy, while other subsequences may be less random. For example, various types of cache lines, cache tags, fingerprints from ASMDS, network routing data, hardware tables or database entries from a branch predictor or hardware prefetcher, to name a few examples.

[0022] At block 204, process 200 selects prefix and suffix data for each data item in a group based on the data content of the item. In some embodiments, the data items may already have identified prefixes and suffixes, in which case block 204 is not necessary. For example, a group of data items may be a group of cache lines that have been provided with cache tags, with some or all of the cache tags used as prefixes. When selecting a prefix, process 200 preferably selects a deterministic subsequence of bits from each item, such as selecting a number of initial bits from the item. The subsequence may be selected from any desired portion of the data item and may include non-sequential bits from the data item, such as a fingerprint or hash calculated from a non-contiguous subsequence of bits from the data item. Alternatively, in some embodiments, a hash or calculation on a desired portion of the data item is used to select or generate the prefix. The suffix data is typically the bits remaining in the data item other than the prefix.

[0023] Next, at block 206, the data items are sorted based on the prefixes. The entire group of data items can be sorted, or the group can be divided into subgroups and the subgroups sorted. In some embodiments, the data items are stored in sorted order of the prefixes, and for these embodiments, block 206 need not be performed. Then, at block 208, a codeword is created that represents the compressed values ​​of all prefixes in a group, as described further below. The codeword is created based on the sorted groups. The codeword can be lossless, that is, it can reproduce every bit of the compressed prefix. If multiple subgroups are sorted, a codeword is typically created for each subgroup.

[0024] At block 210, the suffix data and codeword are stored in memory. Depending on the application, the memory used for storage may be short-term RAM (such as RAM 112 ( Figure 1 )) or storage memory (such as non-volatile memory 120 ( Figure 1 )). The suffix data and code words may also be stored in a database or networked storage device that is not part of the data processing platform that performs the compression and decompression.

[0025] At block 212, when information is requested from the data set, the stored suffix data and codeword are retrieved, and the prefix is ​​decompressed. An example of a suitable decompression process is described below. The decompressed prefix is ​​then paired with the prefix data to provide the complete data item. In some cases, if the desired data item is extracted early in the decompression process, then decompression of the complete codeword is not necessary, as described further below.

[0026] Figure 3 A more detailed process 300 for creating a codeword containing compressed data according to some embodiments is illustrated in flow chart form. Figures 4 to 6 Other examples of compression processes are illustrated in FIG. These processes can be performed by, for example, Figure 1 The compression engine 114 may be executed by a software compression engine, or by other appropriate combination of hardware and software.

[0027] Process 300 begins creating a codeword at block 302. Block 304 provides the encoding table needed to create the codeword by retrieving the codeword from memory, calculating the codeword, or partially calculating the codeword, as further described below.

[0028] For each prefix, in the order provided by the classification, block 306 encodes the prefix by querying a plurality of encoding tables to create a codeword σ containing compressed information representing the values ​​of all prefixes of a set of data items. In this embodiment, block 306 queries the encoding table to obtain a plurality of integers representing the positions of corresponding prefixes in the encoding table.

[0029] like Figure 5 As depicted in , the process can handle variations in classification results as long as the data items are ordered by their prefix values. At block 308, in this embodiment, the results of querying the multiple input tables are used to create a codeword σ by summing the resulting integers as shown. Figure 5 The original input of a set of data items with identified prefixes and suffixes is shown. For simplicity, the depicted data items are only one example with a small number of small data items, and many other arrangements of data items can be stored according to the techniques herein. As shown in block 206 ( Figure 2 ) place, these data items are classified. This classification constitutes a semi-classification with respect to the data items, because no suffix data is used in the classification, only the prefix data. Therefore, the results of the classification can vary in order, such as Figure 5 As shown in the classification outputs A and B of , classification outputs A and B are both valid results of data item classification based on prefixes. Note that two data items sharing the same prefix (101) appear in swapped positions in A and B. More generally, data items with the same prefix can appear in any order relative to each other in the classification output without affecting the correctness of the output. Process 300 is valid even though the order of the input data may vary, such as Figure 5 Those input data in .

[0030] Reference again Figure 3 Although these blocks are described as sequential, the actual order depends on the implementation. When executed by a processor, the process may iterate through each prefix and add the integer value to a running total on each iteration. A more parallel implementation may perform the encoding table lookup in parallel. At block 308, when all integers are summed, the completed summation provides the codeword σ.

[0031] Figure 4 The process for creating codewords containing compressed data using a coding table according to some embodiments is illustrated in table form. The compression code takes an ordered list of k r-bit prefixes and outputs a codeword representing the prefixes. The integer k represents the number of data items in a group, and r represents the number of bits in each prefix. One such coding scheme is depicted, which uses the combined expression The scheme uses an iterator i to enumerate a set of data items in 0, 1, 2, ..., k-2, k-1. As the iterator enumerates, it inserts the value of the prefix of the i-th data item (called p i ) instead of 2 r and i+1 replaces k (if not a zero index, then i will replace k). These partial expressions are then summed to produce the net sum. In this embodiment, the formula for the codeword σ that encodes the k prefixes is given by Equation 1:

[0032] (1)

[0033] The depicted table 400 shows an example of the application of this formula for k=4 and r=log2(3). In this example, the process takes a 4-item prefix and reduces its storage cost from 4log2(3)≈6.34 bits to log2(15)≈3.91 bits. This encoding can be performed by accessing one or more precomputed tables, by computing closed-form expressions in response to compression or decompression requests requiring computed values, or by a combination of the two. Still another alternative is to compute values ​​or portions of values ​​only as needed, but to cache the values ​​after they are computed in associated data or hardware structures for later use. The multiplication computations involved in the computation can also be reduced to a series of equivalent addition, shift, and masking operations.

[0034] For versions using precomputed encoding lookup tables, preferably, a separate lookup table is used for each data item in a group (excluding the 0th data item encoded as the identity function). Based on table 400, four encoding tables 10 to 13 are created to store the values ​​of the combined expressions shown in each of the four columns. These same tables can then also be used for decoding, as described further below.

[0035] In this embodiment, the number of tables is equal to the number of prefixes k. In other embodiments, fewer tables may be used. For example, it is possible to implement Table 10 using implicit encoding, as described below with respect to Figure 10 The tables are generally referred to as tables 10 to 1 k-1 , where each l i is the prefix p of the i-th data item i Coding table. Each l i Save 2 r entries, indexed by a particular pi, resulting in a particular encoding. Specifically, for all integer values ​​of i where 0≤i≤k-1 and where 0≤p i ≤2 r -1 p i All integer values ​​of In this embodiment, an index value representing the prefix position within the table is used to encode the prefix. In other embodiments, other values ​​may be used. For example, the order in which the indexed values ​​appear in the table may be different from the order depicted, in which case the returned index is not necessarily the prefix position.

[0036] Various techniques can be used to improve the efficiency of computing or storing the encoding table. The encoding table can be partially computed on the fly to reduce storage costs. For example, each multiplication by 2 can be implemented as a shift operation, so storing the count of such shifts is more efficient than storing the shifted value. k-1 If two entries in the encoding table store the same value, the process can merge the two entries into a single entry. For example, by selecting appropriate values ​​for r and k, the process can utilize The properties of or similar properties of other encoding schemes can be used to reduce the number of entries required to be stored in each table (e.g., p i =5 and i=2, then Since both are 35). In that case, the process can be solved by converting the tuple p i and a single matrix indexed by i to replace the individual l i , the matrix does not store when i+1 is equal to p i -1 when the value is repeated (ie, the process does not store and Both).

[0037] In some embodiments, for common values ​​of r and k, the size of the encoding table may not exceed one kilobyte. r The order of the k entries is stored, so values ​​of r in the range of about 4 to 8 can be used while still providing most of the space savings and the area overhead of the encoding table is small. In this range of r values, the following Figure 10 The reduction in storage requirements of the exemplary embodiment can be seen in FIG.

[0038] Figure 5 and Figure 6 The sequence diagram illustrates an example compression process using 8-bit data items and employing a 3-bit prefix. Figure 5 The result 500 of a semi-sorting process according to some embodiments is illustrated in diagram form. Four original input data items to the process are depicted, each having 8 bits. A 3-bit prefix and a remaining 5-bit suffix are identified. The two valid variants of the semi-sorted output (e.g., the output of block 206, Figure 2 ) are shown as labeled A and B. As can be seen, two data items in one group have a prefix value of 101. Because sorting rearranges data items according to the relative ordering of their respective prefixes, the two depicted data items with prefix 101 can be interchanged with each other in the sorted output without affecting correctness.

[0039] Figure 6 An exemplary process 600 for creating codewords using a coding table according to some embodiments is illustrated in diagram form. The depicted scenario uses Figure 5A set of data items classified according to output A. The encoding process generally starts with k r-bit prefixes and encodes the prefixes to produce a codeword σ, which is a compressed representation of the prefix. In this case, k is 4, and process 600 uses four encoding tables l3 to l0. The encoding tables can be generated according to the techniques described regarding Figure 4 or using other suitable techniques. The figure shows arrows from each prefix value to the table entries in the corresponding encoding table, indicating that the prefix value is used as an index to the table. If order is not important, the function "nCr" indicates the combinatorial expression "n choose k" (the number of ways to choose k items from a set of n items). As described regarding Figure 3 box 306, an integer value is generated at each corresponding entry. The arrows from the table entries to the adder box indicate that the integer values are summed to produce the codeword σ, which in this example has a value of 261. This codeword value is stored in memory together with a 5-bit suffix value.

[0040] In some embodiments, the encoding tables are provided such that for each prefix, the corresponding prefix index provides a count based on multiple ordered sets of prefix values sorted by a sorting property, where the count is based on the relative position of the prefix in the ordered set. The count of each corresponding prefix index indicates the number of ordered sets of prefix values, where the size of each ordered set is the output of a function applied to the relative position of the prefix value in the ordered set, and all prefix values in the ordered set are less than the corresponding prefix value calculated according to the sorting property. For example, in Figure 6 the embodiment, when examining the i-th prefix, where 0 ≤ i < k, the encoding of the i-th prefix is retrieved by performing a table lookup on l i [prefix]. The returned index value is the count of the sorted list of prefixes of length i + 1 that numerically precedes the first list containing the i-th prefix. For example, if i is 3 and the prefix is 2, then l3[2] stores the count of the list of length 3 + 1 that is sorted in non-increasing order before the list 2, 0, 0, 0. These lists are (1, 1, 1, 1), (1, 1, 1, 0), (1, 1, 0, 0), (1, 0, 0, 0), and (0, 0, 0, 0), for a total of 5. Although in this implementation the sorting property used for the ordered sets is non-increasing, any suitable sorting property that places the prefix values in a defined order can be used.

[0041] Figure 7 Process 700 for decompressing a codeword according to some embodiments is illustrated in flowchart form. Figure 8 Part of process 800 for decompressing a codeword using an encoding table according to some embodiments is illustrated in schematic form. Figure 9 is shown Figure 8 The following description will refer to the diagram of another part 900 of the process as needed. Figures 7 to 9 . Can be decompressed by software engine 116 ( Figure 1 ) or other suitable hardware implementation to perform process 700. The depicted decoding process is an example that can be used given the encoding process described above. For the encoding used above and other encodings, other decoding schemes are also possible.

[0042] At block 702, in response to a request to provide data from a set of data items, process 700 retrieves codewords σ for the set of data items from memory and begins decompression. Stored suffixes for the set of data items are also retrieved, such as along Figure 8 The decoding typically starts with a code word σ, which is a compressed representation of k r-bit prefixes and decodes prefixes starting from the k-1th prefix down to the 0th prefix. The process 700 includes k iterations, four in this example, one for each prefix. Figure 8 and Figure 9 As shown in FIG. 1 , the iteration uses encoding tables 13 to 10 in the order generated by the semi-classification. In this embodiment, the tables used are obtained from Table 1. k-1 The same coding table is used in the coding of Table 10.

[0043] For the first iteration, at block 704, process 700 initializes the integer i to k-1 and initializes the search value σ′ to be equal to the code word σ. Then at block 706, the code word σ is initialized in the encoding table l. i Search for the maximum value that is less than or equal to the search value σ′. Figure 8 This is illustrated in , where Table 13 shows the search results highlighted by searching for the highest value of σ′ less than 261, which is the value 210 found in the last entry.

[0044] At block 708, using this search result, the prefix p i Set equal to the table index of the maximum value found in the search. Figure 9 In the example of , the restored prefix is ​​set to index 111 of the last entry in table 13. The restored prefix value 111 is combined with the stored suffix value 10111 in the order provided by the classification used when encoding the codeword, providing the complete data item shown below table 13.

[0045] Next at block 710, the integer i is set to decrement to proceed to the next iteration. At block 712, the search value σ′ is set equal to the previous search value σ′ minus the largest value found in the previous search iteration that is less than or equal to the previous search value σ′. Figure 9In the example of , this value is shown by the new search value σ′ 2 calculated as 2 61 − 2 10 = 5 1. The source of the value is shown by the arrow feeding into the addition block 902 .

[0046] Then at block 714, in the encoding table 1 i Search for the maximum value that is less than or equal to the new search value σ′. Figure 8 As shown by the highlighted search result 35 in Table 12, the largest table entry is less than 51. At block 716, the restored prefix p i Set to the table index equal to the maximum value found in the search. In the example, the restored prefix p2 is set to the highlighted index value 101 of table 12. This prefix is ​​combined with the stored suffix data to provide the complete data entry for prefix p2, which is 10111111.

[0047] Blocks 710 through 716 are repeated until the last iteration is complete, where the integer i is zero, completing the set of recovered prefixes. Figure 8 and Figure 9 In the example of , the last two iterations showing the resulting values ​​are depicted. Table l1 is searched using the value 16 for σ′1, which is equal to the previous search value 51 from the previous iteration minus the maximum value found, 35. The recovered index p1 is set to the table index value found by searching with 16, which is 101, i.e., index 15 from table l1. Table l0 is then searched using the value 1 for σ′0, which is equal to the previous search value 16 minus the previously found maximum value 15. The recovered index p0 is set to the table index value found by searching with 1, which is 001, i.e., the index of the value 1 found in table l0.

[0048] Note that while the depicted process 700 shows prefix recovery or decompression occurring in its entirety, this is not limiting and the process may be only partially completed if only the value of a prefix recovered early in the process is needed.

[0049] The searches of blocks 706 and 714 may be performed in a linear or parallel fashion for small tables, or alternatively using another suitable search algorithm such as a modified binary search for larger tables. As discussed above, the process may be implemented with a microprocessor, dedicated digital logic, or programmable logic executing program code.

[0050] Figure 10An example of compression performance achieved according to some embodiments is illustrated in graphical form. Graph 1000 shows the performance of the example process described above for multiple data sets, with the reduction in bits per data item on the vertical axis versus the prefix length (r) of bits on the horizontal axis, the number of bits compressed per item. The graph legend identifies different associativity of the data sets. The term "associativity" is used herein as it is commonly used in the art to describe cache sets or similar data sets. That is, each memory address or other data item identifier maps to a possible number of locations within a set or group of data items. The number of possible locations is the associativity. Note that for every power of 2 increase in associativity, approximately one bit is saved per compressed prefix (assuming an appropriate value of r).

[0051] Given a set of k items with r-bit prefixes, the initial space required to store these prefixes is kr bits. This encoding allows the storage process to store all 2 rk However, for every 2 data items r If these values ​​are ordered, then the problem changes from having to store all permutations of the k r-bit numbers to just having to store the combinations. Such combinations are better than all 2 rk Therefore, the technique in this paper can encode these combinations as Therefore, the net savings per group is the net size of k r-bit numbers (kr bits) minus the new size ( Dividing the net savings by the number of terms per group (k), we get the savings in bits per compressed prefix (or term associated therewith), which is Figure 10 is plotted as several instances in , and is generally described by Equation 2:

[0052] (2)

[0053] As can be seen from diagram 1000, for highly associative structures, the storage cost of the prefix can typically be reduced by one to four bits, depending on the associativity set. In ASMDS, data items are typically only eight to several dozen bits long, so saving one to four bits per item is significant. Furthermore, the prefix only needs to be a few bits long to achieve most of the compression benefits. This property is important because the individual encoding lookup tables in the example process herein each have 2r entries, so longer prefixes are expensive in terms of encoding table size.

[0054] The previous technique used a decoding table and an encoding table. If no classification is performed before compression, then all 2 rkEach of these permutations requires an entry in the encoding table. If sorting is done by prefix before compression, only For example, if each entry is 4 bytes in size, then even if r=4 and k=4, there are 3867 entries in the lookup table. At two bytes per entry, each table is approximately 8KB (16KB for both).

[0055] In contrast, some example embodiments herein use k lookup tables, each with 2 r In the case of k = 4 and r = 4, this design uses at most k2 r entries. With two bytes per entry, the total space cost is 2 bytes × 4 × 2^4 = 128 bytes. When r is fixed, the storage cost of this method is roughly linear in σ, but it differs from linear behavior because each lookup table entry uses The order of the units digits is encoded. The storage cost is further reduced by performing the encoding and decoding by computing the lookup table values ​​on the fly. As indicated previously, another possible optimization is to make the final lookup table implicit, since the final lookup table is always That is simply p0, so whatever remains after k-1 iterations of the decompression process is the last prefix. Similarly, during encoding, the final prefix of the encoding process can be encoded into the codeword by adding the zeroth prefix to the sum of integers ( Figure 3 , 308).

[0056] The techniques described herein can be implemented using various combinations of hardware and software. For example, the hardware circuitry may include digital logic, a finite state machine, a programmable logic array (PLA), and the like. The compression and decompression process may be implemented by a microcontroller executing stored program instructions to evaluate the relative timing eligibility of pending commands. In this case, some instructions may be stored in a non-transitory computer memory or computer-readable storage medium for execution by the microcontroller. In various embodiments, the non-transitory computer-readable storage medium includes a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or one or more other non-volatile storage devices. The computer-readable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted and / or executed by one or more processors.

[0057] Figure 1The data processing platform 100 or any portion thereof (such as the processor 110), or other implementations (such as a customized digital logic implementation) may be described or represented by a computer-accessible data structure in the form of a database, database index, or other data structure that can be read by a program and used directly or indirectly to fabricate the integrated circuit. For example, this data structure may be a behavioral-level description or a register-transfer-level (RTL) description of hardware functionality in a high-level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool, which may synthesize the description to generate a netlist comprising a list of gates from a synthesis library. The netlist comprises a set of gates that also represent the functionality of the hardware comprising the integrated circuit. The netlist may then be placed and routed to generate a data set describing the geometry to be applied to a mask. The mask may then be used in various semiconductor fabrication steps to produce the integrated circuit. Alternatively, the database on the computer-accessible storage medium may be a netlist (with or without a synthesis library) or a data set (as desired), or Graphic Data System (GDS) II data.

[0058] While specific embodiments have been described, various modifications to these embodiments will be readily apparent to those skilled in the art. For example, in different embodiments, the type of data in the data set may vary, such as various types of cache lines, cache tags, fingerprints from an ASMDS, network routing data, hardware tables from a branch predictor or hardware prefetcher, or database entries. The mathematical algorithms used to generate the encoding tables may vary. The specific processes for encoding and decoding based on multiple tables may also vary. The methods for providing the encoding tables may also vary, including calculations or partial calculations of table values. Furthermore, while a single codeword is used in the examples herein, other embodiments may generate multiple codewords for a particular data set, depending on the compression algorithm used.

[0059] Therefore, it is intended by the appended claims to cover all modifications of the disclosed embodiments that fall within the scope of this disclosure.

Claims

1. A data processing platform comprising: Memory; a processor coupled to the memory and operable to: Receives a set of data items; selecting suffix data and a prefix for each of the corresponding data items in the set of data items based on data content of the corresponding data item; classifying the set of data items based on the prefix; encoding the prefixes by looking up a plurality of encoding tables to create a codeword σ containing compressed information representing values ​​of all prefixes of the set of data items; storing the suffix data and the code word σ for each of the data items in the memory; decompressing the codeword σ to recover the prefix; and The restored prefix is ​​paired with its corresponding suffix data.

2. The data processing platform according to claim 1, wherein: Each of the plurality of encoding tables includes a plurality of prefix indexes; and Looking up the plurality of encoding tables provides a plurality of integers representing respective prefix indices of the prefixes, and the integers are summed to generate the codeword σ.

3. The data processing platform according to claim 2, wherein: providing the encoding table such that, for each prefix, the corresponding prefix index provides a count based on a plurality of ordered sets of prefix values ​​ordered by the sorting property; and The count is based on the relative position of the prefix in the ordered set.

4. A data processing platform according to claim 3, wherein the count of each corresponding prefix index indicates the number of ordered sets of prefix values, wherein the size of each ordered set is the output of a function applied to the relative position of the prefix value in the ordered set, and all prefix values ​​in the ordered set are less than the value of the corresponding prefix as calculated according to the sorting property.

5. The data processing platform of claim 1, wherein the processor is further operable to compute the plurality of encoding tables at least in part in response to a requirement to decompress the codeword σ.

6. The data processing platform according to claim 1, wherein the plurality of encoding tables comprises k encoding tables 10 to 1 k-1 , the encoding table partially encodes the corresponding prefix among the k prefixes.

7. The data processing platform according to claim 6, wherein the processor is further operable to: When creating the codeword σ, the plurality of encoding tables are accessed to obtain the corresponding partial values ​​of the expression: where p i is the value of the i-th prefix in the order provided by the prefix sort. 8 . The data processing platform according to claim 7 , wherein a plurality of encoding tables contain prefix values ​​and indices that are not arranged in the order of the prefix values.

9. The data processing platform of claim 8, wherein decompressing the codeword σ comprises performing a number of iterations, the number of iterations comprising: For the first iteration, i is initialized to k-1 and the search value σ′ is initialized to be equal to the code word σ; In the coding table i Searching for a maximum value less than or equal to the search value σ′; The restored prefix p i set equal to the table index of said maximum value found in said search; For subsequent iterations, decrementing i and setting the search value σ′ equal to the previous search value σ′ minus the maximum value found in the previous search iteration; In the coding table i searching for the maximum value that is less than or equal to the search value σ′; and The restored prefix p i Set equal to the table index of the maximum value found in the search.

10. The data processing platform of claim 1, wherein the processor is further operable to, when selecting a prefix, select a plurality of non-sequential bits from the corresponding data item.

11. A method for compressing and storing data, comprising: Receives a set of data items; selecting suffix data and a prefix for each of the corresponding data items in the set of data items based on data content of the corresponding data item; classifying the set of data items based on the prefix; encoding the prefixes by looking up a plurality of encoding tables to create a codeword σ containing compressed information representing values ​​of all prefixes of the set of data items; storing the suffix data and the code word σ for each of the data items; decompressing the codeword to recover the prefix; and The restored prefix is ​​paired with its corresponding suffix data.

12. The method according to claim 11, wherein: The plurality of encoding tables each include a number of prefix positions, and The plurality of encoding tables are consulted to provide a plurality of integers representing corresponding prefix positions of the prefixes, and the integers are summed to generate the codeword σ.

13. A method according to claim 12, wherein the encoding table is provided so that for each prefix, the corresponding prefix index provides a count based on a plurality of ordered sets of prefix values ​​sorted by an ordering property, wherein the count is based on the relative position of the prefix in the ordered set.

14. A method according to claim 13, wherein the count of each corresponding prefix index indicates the number of ordered sets of prefix values, wherein the size of each ordered set is the output of a function applied to the relative position of the prefix value in the ordered set, and all prefix values ​​in the ordered set are less than the value of the corresponding prefix as calculated according to the sorting property.

15. The method according to claim 11, wherein the plurality of encoding tables comprises k encoding tables 10 to 1 k-1 , the encoding table partially encodes the corresponding prefix among the k prefixes.

16. The method according to claim 15, further comprising: When creating the codeword σ, the plurality of encoding tables are accessed to obtain the corresponding partial values ​​of the expression: where p i is the value of the i-th prefix in the order provided by the prefix sort.

Images