Semi-sorting compression using encoding and decoding tables

Abstract

The data processing platform, method, and program product perform compression and decompression of a set of data items. For each individual data item in the set of data items, select suffix data and prefixes based on the data content of the individual data item. Sort the set of data items based on the prefixes. Encode the prefixes by querying multiple encoding tables to generate codewords containing compressed information representing the values ​​of all prefixes for the set of data items. Store the codeword and suffix data for each data item in memory. Reconstruct the codeword to recover the prefixes. Pair the recovered prefixes with the individual suffix data.

Description

[Background technology]

[0001] Semi-sorting has been described for use in compressing fingerprints in cuckoo filters and d-left counting Bloom filters, two important approximate set membership data structures (ASMDS) that are prevalent in networking hardware and software, among other applications (e.g., database systems, genome sequencing, and file systems). A fingerprint is a short hash that encodes the presence of an item in the set that the ASMDS approximately represents. Typically, fingerprints are stored in buckets (similar to cache sets) in the ASMDS. The position of a fingerprint within a bucket does not change its meaning.

[0002] Previous work has ordered these fingerprints by their prefixes and replaced the prefixes with code words. To encode and decode the fingerprints, previous work has used a set of encoding and decoding tables. In practice, however, the growth in size of these tables relative to the associativity limits the associativity of buckets to 4 or less, since for an associativity of 4, the size of the lookup tables is several kilobytes. Thus, a solution is desired to reduce the size of these tables and improve their storage in hardware caches or other memories. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of a data processing platform including compression according to some embodiments. [Diagram 2] FIG. 2 is a flow diagram of a process for compressing and decompressing data according to some embodiments. [Diagram 3] FIG. 4 is a flow diagram of a more detailed process for forming codewords including compressed data according to some embodiments. [Figure 4] 1 is a table illustrating a process for forming a codeword containing compressed data using an encoding table according to some embodiments. [Diagram 5] FIG. 1 illustrates a semi-sorting process, according to some embodiments. [Figure 6] FIG. 2 illustrates an example of forming a codeword using an encoding table, according to some embodiments. [Figure 7] FIG. 4 is a flow diagram of a process for recovering a codeword according to some embodiments. [Figure 8] FIG. 2 illustrates a portion of a process for recovering a codeword using an encoding table according to some embodiments. [Figure 9] FIG. 9 illustrates a further portion of the process of FIG. 8. [Figure 10] 1 is a chart illustrating an example of compression performance obtained according to some representative embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] In the following description, the use of the same reference numbers in different figures indicates similar or identical items. Unless otherwise specified, the term "coupled" and its associated verb forms include both direct and indirect electrical connections by means well known in the art, and unless otherwise specified, any description of a direct connection also includes the meaning of an alternative embodiment using an appropriate form of indirect electrical connection.

[0005] The data processing platform includes a memory and a processor coupled to the memory, the processor capable of compressing and decompressing a set of data items. The processor receives the set of data items and selects suffix data and prefixes for each individual data item in the set of data items based on data content of the individual data item. The set of data items is sorted based on the prefixes. The prefixes are encoded by querying a plurality of encoding tables to form codewords containing compressed information representative of values ​​of all prefixes for the set of data items. The suffix data for each of the data items and the codeword is stored in the memory. The codeword is restored to recover the prefixes and paired with their respective suffix data.

[0006] The method compresses and stores data items: receives a set of data items and selects suffix data and prefixes for each individual data item in the set of data items based on the data content of the individual data item; sorts the set of data items based on the prefixes; encodes the prefixes by consulting multiple encoding tables to form codewords containing compressed information representing the values ​​of all prefixes for the set of data items; stores the codeword and suffix data for each of the data items; recovers the codeword to recover the prefixes; and pairs the recovered prefixes with their individual suffix data.

[0007] A tangible, non-transitory computer readable medium carries a program product executable by at least one processor to compress and decompress data items: receive a set of data items and select suffix data and prefixes for each individual data item in the set of data items based on data content of the individual data item; sort the set of data items based on the prefixes; encode the prefixes by consulting a plurality of encoding tables to form codewords containing compressed information representative of the values ​​of all prefixes for the set of data items; store the codewords and suffix data for each of the data items in a memory; recover the codewords to recover the prefixes; and pair the recovered prefixes with their respective suffix data.

[0008] FIG. 1 shows a block diagram of a data processing platform including compression, according to some embodiments. Data processing platform 100 may be a platform that needs to compress a set of data items with high entropy that may be arbitrarily reordered. For example, the techniques herein have many applications, such as compressing software and hardware cache tags and fingerprints in approximate set membership data structures. In general, data processing platform 100 performs compression and decompression on a set of items of fixed length. The techniques are most useful for uniform random items (i.e., maximum information entropy) whose order can be permuted. Applications may vary from networking, software systems, and hardware caches.

[0009] The data processing platform 100 communicates with a network or host system 10 to fulfill requests for data storage and retrieval or to perform operations such as encryption key retrieval or caching for the host system. The data processing platform 100 includes a processor 110 in this example in communication with a random access memory (RAM) 112, which may be internal or external to the processor 110. The RAM 112 may be, for example, part of an integrated cache of the processor 110 or may be in the main memory of the platform 100. The processor 110 also communicates with a non-volatile memory 120, which is a tangible, non-transitory computer readable medium (e.g., flash memory or a hard drive) that holds computer program code, such as compression and decompression program code 122.

[0010] RAM 112 holds data used in the compression techniques described below, such as codeword σ and suffixes 0 through k−1 (there are k number of suffixes, each associated with one of the k data items). Such data may also be stored in non-volatile memory. Once program code 122 is loaded from non-volatile memory 120, the compression and decompression program code resides in RAM 112. This is indicated by loaded compression engine 114 and loaded decompression engine 116. Other data for use with the techniques described herein, such as data items to be compressed and encoding tables, may also be held in RAM 112 or non-volatile memory 120.

[0011] In this embodiment, the data processing platform includes a processor 110 that executes program code instructions, although in other embodiments some or all of the compression functions may be implemented in hardware, such as digital logic in a networking ASIC, a CPU, a GPU, or programmable logic within an FPGA.

[0012] 2 illustrates a flow diagram of a process 200 for data compression and decompression according to some embodiments. A set of data items is prepared for compression (block 202). The data items are typically fixed-length items and, as previously discussed, often have high entropy. Some subsequences of bits within the data items may have high entropy, while other subsequences may have less randomness. For example, various types of cache lines from an ASMDS, cache tags, fingerprints, network routing data, hardware tables from a branch predictor or hardware prefetcher, or database entries, to name a few.

[0013] At block 204, process 200 selects prefix and suffix data for each data item in the set based on the data content of the item. In some embodiments, the data items may already have prefixes and suffixes identified, in which case block 204 is not necessary. For example, the set of data items may be a set of cache lines for which a cache tag has already been given, and a partial or complete tag is used as the prefix. When selecting the prefix, process 200 preferably selects a deterministic subsequence of bits from each item (e.g., the first number of bits from the item). The subsequence may be selected from any desired portion of the data item, and may include non-contiguous bits from the data item (e.g., a fingerprint or hash calculated from non-contiguous bit subsequences from the data item). Alternatively, a hash or calculation on the desired portion of the data item is used in certain embodiments to select or generate the prefix. The suffix data is typically the bits remaining in the data item excluding the prefix.

[0014] Next, in block 206, the data items are sorted based on the prefixes. Sorting may be performed on the entire set of data items, or the set may be divided into subsets and the subsets sorted. In some embodiments, the data items are stored in an order that follows the ordering of the prefixes, and for these embodiments, block 206 need not be performed. Next, in block 208, codewords are formed that represent the compressed values ​​of all the prefixes in the set, as described below. Codewords are formed based on the sorted set. The codewords may be non-lossy, i.e., capable of recovering all the bits of the compressed prefixes. If multiple subsets are sorted, a codeword is typically formed for each subset.

[0015] At block 210, the suffix data and codewords are stored in memory. Depending on the application, the memory used for storage may be short term RAM (e.g., RAM 112 (FIG. 1)) or storage memory (e.g., non-volatile memory 120 (FIG. 1)). The suffix data and codewords may also be stored in a database or network attached storage device that is not part of the data processing platform that performs the compression and decompression.

[0016] When information from the data set is requested, the stored suffix data and codeword are retrieved to recover the prefix (block 212). An example of one suitable recovery process is described below. The recovered prefix is ​​then paired with the prefix data to obtain the complete data item. In some cases, as described below, it may not be necessary to recover the complete codeword if the desired data item was extracted early in the recovery process.

[0017] Figure 3 shows a flow diagram of a more detailed process 300 for forming codewords including compressed data, according to some embodiments. Further examples of compression processes are shown in Figures 4-6. These processes may be performed by a software compression engine (e.g., compression engine 114 of Figure 1) or by other suitable combinations of hardware and software.

[0018] Process 300 begins with the formation of a codeword at block 302. At block 304, the encoding tables necessary to form the codeword are provided by retrieving them from memory, computing them, or partially computing them, as described below.

[0019] For each prefix, in the order provided by the sorting, block 306 encodes the prefix by querying a number of encoding tables to form a codeword σ that includes compressed information representing the values ​​of all prefixes for the set of data items, in this embodiment, block 306 queries the encoding tables to obtain a number of integers that represent respective prefix positions within the encoding tables.

[0020] As long as the data items are ordered via their prefix values, the process may handle variations in the sorting results, as shown in FIG. 5. In block 308, the results of querying the multiple input tables are used to form the codeword σ, in this embodiment by summing the resulting integers, as shown. FIG. 5 shows a raw input of a set of data items for which prefixes and suffixes have been identified. The data items shown are merely an example with a small number of small data items for simplicity, and many other configurations of data items may be stored in accordance with the techniques herein. These data items are sorted as described in block 206 (FIG. 2). This sort constitutes a semi-sort on the data items because no suffix data is used in the sorting, only prefix data. Thus, the results of the sort may vary in order. This is shown by sort outputs A and B in FIG. 5, both of which are legitimate results of a prefix-based sort on the data items. Note that two data items sharing the same prefix (101) appear in swapped positions in A and B. More generally, data items with the same prefix may appear in any order in the output of a sort relative to one another, without affecting the correctness of the output. Process 300 is effective despite possible variations in the order of the input data (e.g., those shown in FIG. 5).

[0021] Referring again to Figure 3, the blocks are shown to be sequential, although the actual order is implementation dependent. When performed by a processor, the process may iterate through each prefix, adding an integer value to the accumulated sum at each iteration. For more parallel implementations, the encoding table lookups may be performed in parallel. Once all integers have been summed (block 308), the completed sum yields the codeword σ.

[0022] Figure 4 shows a table of a process for forming codewords containing compressed data using an encoding table, according to some embodiments. Compressive encoding takes an ordered list of k r-bit prefixes and outputs codewords that represent them. The integer k represents the number of data items in the set, and r represents the number of bits in each prefix. One such encoding scheme is shown. It is based on the combinatorial representation

number

number

[0023] Illustrated table 400 shows an example of the application of this formula following k=4 and r=log2(3). In this example, the process takes four data item prefixes and reduces their storage cost from 4log2(3) ~ 6.34 bits to log2(15) ~ 3.91 bits. This encoding can be done by accessing one or more pre-computed tables, by computing closed form expressions in response to a compression or decompression request that requires a computed value, or by a combination of these. Yet another alternative is to only compute the value or sub-value when needed, but cache it after it is computed in an associated data or hardware structure for later use. It is also possible to reduce the multiplication calculations involved in the calculation to a series of equivalent add, shift and masking operations.

[0024] In the version using pre-computed encoding lookup tables, a separate lookup table is preferably used for each data item in the set (except for the 0th data item, whose encoding is the identity function). From table 400, four encoding tables l0-l3 are formed to hold values ​​for the combinatorial expressions shown in each of the four columns. These same tables can also be used for decoding, as described below.

[0025] In this embodiment, the number of tables is equal to the number of prefixes k. In other embodiments, fewer tables may be used. For example, table l0 may be implemented using implicit encoding, as will be further described below with respect to FIG. 10. The tables may be generally represented as tables l0-l. k-1 It is called. Each l i is the prefix p of the i-th data item i Here is the encoding table for each l i is 2 r entries, and a specific p i(indicating a particular encoding). Specifically,

number

[0026] Various techniques may be used to increase the efficiency of computing or storing the encoding tables. The encoding tables may be partially computed on the fly to reduce their storage cost. For example, all multiplications by 2 may be implemented as shift operations, and it is therefore more efficient to store counts of such shifts rather than the shifted values. The encoding tables l0 through l k-1 In providing x, if two entries in the encoding table store the same value, the process may merge the two entries into a single entry. For example, by selecting appropriate values ​​for r and k, the process may

number

number

number

[0027] In some embodiments, the encoding table may not exceed kilobytes in size for common values ​​of r and k. The process simply r Since only k entries need to be stored, values ​​of r in the range of about 4 to 8 may be used while still achieving most of the space savings and with a small area overhead for the encoding table. The reduced storage requirements for an exemplary embodiment for values ​​of r in this range can be seen in Figure 10 below.

[0028] The sequences of Figures 5 and 6 show an example compression process using 8-bit data items with a 3-bit prefix. Figure 5 shows the result 500 of a semi-sorting process according to some embodiments. Shown are four raw input data items (each 8 bits) to the process. The 3-bit prefix and the remaining 5-bit suffix are identified. Two legal variations of the semi-sorted output (e.g., the output of block 206 in Figure 2) are shown as indicated A and B. As can be seen, the prefix value of two data items in the set is 101. In the sorted output, the two illustrated data items with prefix 101 can be substituted for each other without affecting accuracy, since the sorting rearranges the data items subject to the relative ordering of their respective prefixes.

[0029] FIG. 6 illustrates an exemplary process 600 for forming codewords using encoding tables, according to some embodiments. The illustrated scenario uses the set of data items from FIG. 5 (sorted by output A). The encoding process generally begins with k r-bit prefixes and encodes the prefixes to generate a codeword σ that is a compressed representation of the prefixes. In this scenario, k is 4, and the process 600 uses four encoding tables l3-l0. The encoding tables may be generated by the technique described with respect to FIG. 4, or by any other suitable technique. The diagram shows arrows from each prefix value to the table entry in the respective encoding table for that prefix. The prefix value is shown to be used as an index into the table. The function “nCr” represents the combinatorial representation “n choose k” (the number of ways to select k items from a group of n items when order does not matter). An integer value results in each respective entry, as described with respect to block 306 of FIG. 3. The arrows from the table entries to the adder blocks represent the integer values ​​being summed to generate the codeword σ (which in this example has a value of 261). This codeword value is stored in memory along with a 5-bit suffix value.

[0030] In some embodiments, the encoding table is provided such that for each prefix, an individual prefix index gives a count based on a plurality of ordered sets of prefix values ordered by an ordering characteristic, and the count is based on the relative position of the prefix within the ordered set. The count for each individual prefix index indicates the number of ordered sets of prefix values, and the size of each ordered set is the output of a function applied to the relative position of the prefix value within the ordered set, and all prefix values within the ordered set are less than the value of the individual prefix calculated by the ordering characteristic. For example, in the embodiment of FIG. 6, when testing the i-th prefix (0 ≦ i < k), the encoding of the i-th prefix is retrieved by performing a table lookup on l i [prefix]. The returned index value is the count of the number of sorted lists of length i + 1 of numerically preceding prefixes of the first list containing the i-th prefix. For example, if i is 3 and the prefix is 2, then l3[2] stores the count of the number of lists of length 3 + 1 ordered in non-increasing order preceding the list 2, 0, 0, 0. These are (1, 1, 1, 1), (1, 1, 1, 0), (1, 1, 0, 0), (1, 0, 0, 0), and (0, 0, 0, 0), for a total of five. In this embodiment, the ordering characteristic used with the ordered set is non-increasing, but any suitable ordering characteristic that arranges the prefix values in a defined order may be used.

[0031] FIG. 7 illustrates a flow diagram of a process 700 for recovering a codeword, according to some embodiments. FIG. 8 illustrates a portion of a process 800 for recovering a codeword using an encoding table, according to some embodiments. FIG. 9 illustrates a further portion 900 of the process of FIG. 8. The following description refers to FIGS. 7-9 as necessary. The process 700 may be performed by software recovery engine 116 (FIG. 1) or other suitable hardware embodiments. The illustrated decoding process is one example that may use the predetermined encoding process described above. Other decoding schemes are possible for the above encodings or other encodings.

[0032] In response to a request to obtain data from a set of data items, at block 702, process 700 begins the decompression by retrieving from memory the codeword σ for the set of data items. As shown along the top of FIG. 8, the stored suffixes for the set of data items are also retrieved. The decoding generally begins with the codeword σ, which is a compressed representation of k r-bit prefixes, and performs prefix decoding starting from the k-1th prefix through the 0th prefix. Process 700 includes k iterations (four in this example, one per prefix). The iterations use encoding tables l3-l0 (as shown in FIGS. 8 and 9) in the order generated by semi-sorting. In this embodiment, the tables used are table l k-1 This is the same encoding table used to encode table l0.

[0033] In the first iteration, the process 700 in block 704 initializes an integer i to k1 and a search value σ′ to a value equal to the codeword σ. Then, in block 706, the encoding table l is i is searched to find the largest value less than or equal to the search value σ'. This is illustrated in Figure 8, which shows table l3 with the search results highlighted from the search for the highest value less than σ' of 261. This is the value 210 seen in the last entry.

[0034] This search result is used in block 708. In block 708, the regenerated prefix p i set l3 equal to the table index of the maximum value found in the lookup. In the example of Figure 9, set the regenerated prefix to the index of the last entry in table l3, 111. The regenerated prefix value of 111 is combined with the stored suffix value of 10111 in the order obtained by the sorting used to encode the codeword to obtain the complete data item shown at the bottom of table l3.

[0035] Then, in block 710, the integer i is set decremented to proceed to the next iteration. In block 712, the search value σ' is set to a value equal to the previous search value σ' minus the maximum value found in the previous search iteration that was less than or equal to the previous search value σ'. In the example of Figure 9, this value is indicated by the new search value σ2', calculated as 261-210=51. The source of the value is indicated by the arrow feeding into summation block 902.

[0036] Next, in block 714, the encoding table l i , for the largest value less than or equal to the new search value σ′. This is illustrated in FIG. 8 by the highlighted search result in table l2 of 35 (the largest table entry less than 51). In block 716, the regenerated prefix p i to a value equal to the table index of the maximum value found in the search. In the example, the regenerated prefix p2 is set to the highlighted index value 101 for table l2. This prefix is ​​combined with the stored suffix data to obtain the complete data item for prefix p2, which is 10111111.

[0037] Blocks 710-716 are repeated until the final iteration is completed with integer i at zero, completing the set of regenerated prefixes. In the example of Figures 8-9, the final two iterations are shown, showing the resulting values. Table l1 is searched with σ1' value 16, which is equal to the previous lookup value of 51 from the previous iteration minus the maximum value found of 35. The regenerated index p1 is set to the table index value found in the lookup with 16 (which is 101, index 15 from table l1). Table l0 is then searched with σ0' value 1, which is equal to the previous lookup value of 16 minus the maximum value found so far of 15. The regenerated index p0 is set to the table index value found in the lookup with 1 (which is 001, index of value 1 found in table l0).

[0038] It should be noted that while the illustrated process 700 shows prefix regeneration or restoration occurring completely, this is not a limitation and the process may only be partially complete if values ​​are needed only from prefixes that were regenerated earlier in the process.

[0039] The searches of blocks 706 and 714 may be performed in a linear or parallel manner on a small table, or alternatively on a larger table using another suitable search algorithm, such as a modified binary search. As previously mentioned, the processes may be implemented using a microprocessor executing program code, using application specific digital logic, or using programmable logic.

[0040] FIG. 10 shows a chart of example compression performance obtained by some embodiments. Chart 1000 shows the performance of the above-described example process for multiple data sets, showing bit reduction per data item (vertical axis) versus prefix length in bits (r) (number of compressed bits per item) (horizontal axis). The chart symbols identify different associativity of the data sets. The term "associativity" is used herein as it is commonly used in the art to describe a cache set or similar set of data. That is, each memory address or other data item identifier is mapped to a possible number of locations within the set or group of data items. The number of possible locations is the associativity. Note that for every power of 2 increase in associativity, approximately one bit is saved per compressed prefix (assuming a suitable value for r).

[0041] Assuming that k-itemsets are accompanied by r-bit prefixes, the initial space for storing these prefixes is kr bits. With such an encoding, the storage process can store all 2 rk permutations can be encoded, but only exactly 2 permutations can be encoded per data item. r If there are distinct values, and these values ​​are ordered, the problem evolves from needing to store all permutations of the kr-bit number to simply needing to store combinations.

number

number

number

number

[0042] As can be seen from chart 1000, for highly associative structures, the storage cost of the prefix can often be reduced by 1 to 4 bits, depending on the set associativity. In an ASMDS, data items are often only 8 to tens of bits in length, so a savings of 1 to 4 bits per item is significant. Moreover, the prefix only needs to be a few bits in length to realize most of the compression benefit. This property is important because the individual encoding lookup tables in the example process herein each require 2 r entries, longer prefixes come at a cost in terms of encoding table size.

[0043] Previous work has used one decoding table and one encoding table. Without sorting before compression, rk Each permutation requires an entry in the encoding table. If we use prefix sorting before compression,

number

[0044] In contrast, some embodiments herein use k lookup tables, each of which has a length of 2 r For k=4 and 4=4, such a design has at most k2 r entries. At 2 bytes per entry, the overall space cost is 2 bytes * 4 * 2^4 = 128 bytes. The storage cost of this approach scales approximately linearly with k when r is fixed, but deviates from linear behavior because each lookup table entry

number

number

[0045] The techniques described herein may be implemented using various combinations of hardware and software. For example, hardware circuits may include digital logic, finite state machines, programmable logic arrays (PLAs), etc. The compression and decompression process may be implemented by a microcontroller executing stored program instructions to evaluate the relative timing of pending commands. In this case, some of the instructions may be stored in a non-transitory computer memory or computer readable storage medium for execution by the microcontroller. In various embodiments, the non-transitory computer readable storage medium includes a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, or other non-volatile memory device or device. The computer readable instructions stored in the non-transitory computer readable storage medium may be source code, assembly language code, object code, or other instruction formats that can be interpreted and / or executed by one or more processors.

[0046] The data processing platform 100 of FIG. 1, any portion thereof (e.g., processor 110, etc.), or other embodiments (e.g., customized digital logic embodiments, etc.) may be described or represented by a computer-accessible data structure in the form of a database, a database index, or other data structure that can be read by a program and used directly or indirectly to manufacture an integrated circuit. For example, the data structure may be a behavioral or register transfer level (RTL) description of the hardware functionality in a high-level design language (HDL), such as Verilog or VHDL. The description may be read by a synthesis tool that may synthesize the description to generate a netlist that includes a list of gates from a synthesis library. The netlist includes a set of gates that represent the functionality of the hardware that comprises the integrated circuit. The netlist may then be placed and routed to generate a data set that describes the geometric shapes to be applied to a mask. The mask may then be used in various semiconductor manufacturing steps to manufacture the integrated circuit. Alternatively, the database on the computer-accessible storage medium may be a netlist (with or without a synthesis library) or a data set, if desired, Graphics Data System (GDS) II data.

[0047] Although specific embodiments have been described, various modifications to these embodiments will be apparent to those skilled in the art. For example, the types of data in the data set may be different in different embodiments, such as various types of cache lines, cache tags, fingerprints from ASMDS, network routing data, hardware tables from a branch predictor or hardware prefetcher, or database entries. The mathematical algorithms used to generate the encoding tables may be different. The specific processes for encoding and decoding based on multiple tables may also be different. The methods of providing the encoding tables may also be different, including computation or partial computation of table values. Additionally, while the examples herein use a single codeword, other embodiments may generate multiple codewords for a particular data set, depending on the compression algorithm used.

[0048] It is therefore intended by the appended claims to cover all modifications of the disclosed embodiments that fall within the scope of this disclosure.

Claims

1. 1. A data processing platform, comprising: Memory, a processor coupled to the memory, The processor, receiving a set of data items; selecting, for each individual one of the data items, suffix data and a prefix based on the data content of the individual one of the data items; Sorting the set of data items based on the prefix; and encoding the prefixes by querying a plurality of encoding tables and summing integers obtained from the plurality of encoding tables to generate a codeword σ containing compressed information representative of values ​​of all prefixes for the set of data items; storing the suffix data and the codeword σ for each of the data items in the memory; recovering said codeword σ to recover said prefix; pairing said recovered prefixes with respective suffix data; [0023] Data processing platform.

2. each of the plurality of encoding tables includes a number of prefix positions; obtaining the plurality of integers representing prefix positions of each of the prefixes by querying the plurality of encoding tables; The data processing platform of claim 1.

3. the encoding table is arranged such that for each prefix, a respective prefix index representative of the prefix position provides a numerical value based on a plurality of ordered sets of prefix values ​​ordered by an ordering property; the numerical value being based on the relative position of the prefix within the ordered set; The data processing platform of claim 2.

4. the number for each prefix index indicates a number of ordered sets of prefix values, the size of each ordered set being the output of a function applied to the relative position of the prefix value within the ordered set, and all prefix values ​​in the ordered set are less than the value of the individual prefix as calculated by the ordering property; The data processing platform of claim 3.

5. the processor is operable to at least partially compute the plurality of encoding tables in response to a request to recover the codeword σ. The data processing platform of claim 1.

6. The plurality of encoding tables includes a number k of encoding tables l ０ ~l ｋ－１ wherein the encoding table partially encodes each prefix of the k prefixes. The data processing platform of claim 1.

7. The processor, when generating the codeword σ, the method is operable to access the plurality of encoding tables to obtain respective sub-values ​​of the expression [0010] σ is the codeword, and p ｉ is the value of the i-th prefix in the order obtained by sorting the set of data items based on the prefixes; The data processing platform of claim 6.

8. the plurality of encoding tables includes the prefix values ​​and prefix indexes not arranged in order of the prefix values; The data processing platform of claim 7.

9. Recovering the codeword σ comprises performing a number of iterations; The plurality of iterations includes: in a first iteration, initializing i to k−1 and a search value σ′ to a value equal to said codeword σ; Encoding Table ｉ to search for a maximum value equal to or less than the search value σ′; The prefix p ｉ setting a value equal to the table index of the maximum value found in said search; in a subsequent iteration, decrementing i and setting said search value σ′ equal to the previous search value σ′ minus the maximum value found in the previous search iteration; Encoding Table ｉ to search for a maximum value equal to or less than the search value σ′; The prefix p ｉ setting x equal to a table index of the maximum value found in the search. The data processing platform of claim 8.

10. the processor sorts the data items using only the prefixes. The data processing platform of claim 1.

11. 1. A method for compressing and storing data, comprising the steps of: receiving a set of data items; selecting, for each individual data item in the set of data items, suffix data and a prefix based on the data content of the individual data item; Sorting the set of data items based on the prefix; and encoding the prefixes by querying a plurality of encoding tables and summing integers obtained from the plurality of encoding tables to generate a codeword σ containing compressed information representative of values ​​of all prefixes for the set of data items; storing the suffix data and the codeword σ for each of the data items; recovering the codeword to recover the prefix; Pairing the recovered prefixes with respective suffix data. method.

12. each of the plurality of encoding tables includes a number of prefix positions; obtaining the plurality of integers representing prefix positions of each of the prefixes by querying the plurality of encoding tables; 12. The method of claim 11.

13. the encoding table is arranged such that for each prefix, a respective prefix index representative of the prefix position provides a numerical value based on a plurality of ordered sets of prefix values ​​ordered by an ordering property; the numerical value being based on the relative position of the prefix within the ordered set; 13. The method of claim 12.

14. the number for each prefix index indicates a number of ordered sets of prefix values, the size of each ordered set being the output of a function applied to the relative position of the prefix value within the ordered set, and all prefix values ​​in the ordered set are less than the value of the individual prefix as calculated by the ordering property; 14. The method of claim 13.

15. The plurality of encoding tables includes a number k of encoding tables l ０ ~l ｋ－１ wherein the encoding table partially encodes each prefix of the k prefixes.

12. The method of claim 11.

16. When generating the codeword σ, the method further includes accessing the plurality of encoding tables to obtain respective subvalues ​​of the following equation: [0025] σ is the codeword, and p ｉ is the value of the i-th prefix in the order obtained by sorting the set of data items based on the prefixes; 16. The method of claim 15.

Images