Implementation method, system and device for top-speed table item compression

Abstract

The invention discloses an implementation method, system and device for top-speed table item compression, and belongs to the technical field of data compression. The implementation method comprises the following steps: generating a public template and compressed data through bit level statistics, and compressing and decompressing the data based on template matching; a manual template input and automatic template training mechanism is supported; hardware pipelined table item compression and decompression are supported, and the time delay is in a nanosecond level. The invention provides a top-speed table item compression system and device. The top-speed table item compression system comprises a template training module, a table item compression module and a table item decompression module. The compression and decompression method based on bit-level deduplication and template matching is suitable for real-time hardware platforms such as ASIC and FPGA, can be widely applied to a CPU, a GPU, a DPU, a mobile terminal SoC and various heterogeneous computing chips, solves the problem of delay mismatching of a traditional software compression algorithm on the hardware platforms, greatly reduces table item storage space, and improves the compression efficiency. And the data storage and transmission efficiency in a high-real-time scene is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, and in particular to a super-low latency lossless compression method based on bit-level template matching, which is suitable for efficient data storage and transmission in a hardware chip environment. BACKGROUND

[0002] Data compression is one of the core technologies of computer science, aiming to reduce the storage space and transmission bandwidth occupied by data to improve system efficiency. Its applications cover storage systems, communication networks, artificial intelligence, multimedia processing, etc., such as file compression, real-time video streaming, database optimization, etc. According to the integrity of the compressed data, data compression can be divided into two categories: lossy compression and lossless compression:

[0003] Lossy compression: allows data to be partially lost during compression and decompression to greatly reduce data volume, suitable for scenarios with high tolerance to precision (such as images, audio, video). Typical algorithms include JPEG (image), MP3 (audio), H.264 (video), etc.

[0004] Lossless compression: ensures that the data before and after compression is completely consistent, suitable for scenarios with strict requirements for integrity such as text, program code, medical data, etc. Common algorithms include Huffman coding based on statistical coding, arithmetic coding, and LZ77, LZ78, DEFLATE (Gzip) based on repetition elimination.

[0005] Existing compression technologies are relatively mature at the software level, but there is a fundamental conflict between their design philosophy and hardware execution characteristics, and they face problems such as low compression efficiency, high latency, and large resource occupation in chip integration scenarios. Especially in the fields of AI inference, 5G communication, edge computing, etc., which have strict requirements for real-time performance and energy efficiency, a new compression method is needed to ensure compression rate while achieving hardware-friendly nanosecond-level compression and high parallelization support. SUMMARY

[0006] Based on the above status, the main purpose of the present application is to provide an implementation method, system and device for ultra-fast table compression, which realizes nanosecond-level compression / decompression through bit-level template matching and compressed data extraction, and is compatible with multiple hardware chip architectures.

[0007] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0008] The first aspect of the present application provides a method for implementing ultrafast table entry compression, a table entry compression data structure comprising a template and compressed data, the method comprising: performing binary bit pattern comparison on entries of an original data table, quantifying bit-level similarity distribution characteristics between entries to generate a template. The template comprises common bit data and coordinate annotation information, wherein the coordinate annotation information determines the positions of common bit positions and difference bit positions through a binary mask or coordinate index; in response to a write table request of a to-be-stored table entry, determining a matching template of the to-be-stored table entry to determine whether to perform table entry compression according to a matching state of the to-be-stored table entry; in the case of performing table entry compression, extracting corresponding bit data based on the positions of difference bit positions of the coordinate annotation information according to the matching template, and splicing to generate compressed data in the order of original positions, and storing the compressed data into the data table; in the case of not performing table entry compression, storing the original table entry into the data table; storing the compressed data or the original table entry into a determined storage position of the data table, and recording an index of the compressed data or the original table entry by using a mapping table. The mapping table adopts an address mapping scheme or a direct mapping scheme to associate the index of the original table entry to the index of the data table and / or the template table, to realize fast addressing of table entry compression and decompression; in response to a read table request of a table entry, obtaining difference data and a template of the to-be-read table entry according to the data index and / or the template index recorded by the mapping table, determining the position of the difference data in the original table entry according to the difference bit positions of the coordinate annotation information of the template; determining the position of the common data in the original table entry according to the common bit positions of the coordinate annotation information of the template; and determining the to-be-read table entry by splicing the difference bit data position, the common bit data position, the compressed data in the data table and the common bit data of the coordinate annotation information.

[0009] Preferably, the method further comprises the following decompression step:

[0010] According to the table entry index, the mapping table and / or the data table are read. The mapping table is used to determine whether it is compressed data, the template index corresponding to the compressed data and the data index; and the data table is used to store compressed data or original data. If it is non-compressed data, the data table is read in a chain segment to splice and generate original data; if it is compressed data, the compressed data and the matching template are obtained based on the data index and the template index, the data source of the bit position is determined according to the coordinate annotation information, the data of the common bit position comes from the template, and the data of the difference bit position comes from the data table.

[0011] Preferably, the method further comprises the following scheme:

[0012] The address mapping scheme comprises: constructing an independent mapping table to record the association relationship of the index of the original table entry or the compressed data, the template index and the compression marker; and accelerating decompression addressing by reading the mapping table and the template table in parallel.

[0013] Preferably, the method further comprises the following steps:

[0014] The direct mapping scheme comprises: merging the data table with a mapping table, wherein the entries of the mapping table contain a compression flag, a template index and a data segment; and associating the uncompressed data with a reserved segment and an extended segment through a linked list pointer, wherein the reserved segment stores the first segment data and the extended segment is dynamically allocated.

[0015] In the second aspect of the present application, a template generation method is provided, which comprises: automatically training templates based on a similarity search algorithm, wherein in the initial stage of the automatic training, a set of spare templates is emptied or an experience template is filled according to previous experience of the administrator, and the public bit in the coordinate annotation information in the experience template is empty or not full, and then according to each table entry of the original data table, the matching state of the table entry in the set of spare templates is determined, and the public bit in the coordinate annotation information and the public data are expanded in the matched template.

[0016] Preferably, the method further comprises the following steps:

[0017] The automatic training function of the template training module is realized by a hardware circuit, which comprises: a bit-level statistical circuit based on a greedy algorithm, a local sensitive hashing algorithm and an unsupervised clustering algorithm; and a parallel processing unit for incremental training and template replacement.

[0018] In the third aspect of the present application, a template matching method is provided, which comprises: in response to a storage request of a table entry, comparing the table entry data with templates in a template table bit by bit; if the public bit data is inconsistent, replacing the template to be matched with the next one in the matching template table; if all are consistent, the template matching hits, and the template is a matching template; in the case of template matching hit, determining the corresponding compressed data according to the matching template and storing it into the data table. When it is not hit, the original data is stored into the data table.

[0019] In the fourth aspect of the present application, a template replacement method is provided, which comprises: traversing the table entries in a mapping table referring to a replaced template, and in the case that there is free space in the data table, restoring the original table entry according to the replaced template and the corresponding difference data, and storing the original table entry as a new table entry of the data table. If the replaced template reference count in the template matching table is zero, the replaced template is deleted and a high reference template in the spare template table is added. The traversing the table entries in the mapping table referring to the replaced template comprises actively triggering the data table scanning and passively triggering when reading the table entry.

[0020] In a fifth aspect, the application provides a processing device for ultrafast entry compression, comprising: a template training module, supporting manual generation of templates and automatic training based on algorithms, including an entry data storage unit, a training algorithm execution unit, a backup template storage unit, and a template replacement execution unit; an entry compression module, including a template matching unit, a compressed data extraction unit, a data table, and a template table, for performing template matching, extracting compressed data, and data storage; the template matching unit includes a template table to be matched and a matching processing unit; the data table includes compressed data of template matching hit entries and original data of template matching misses; the template table includes templates corresponding to hit data of template matching; and an entry decompression module, for reading compressed data in the data table according to a data index, reading a matching template of the compressed data according to a template index, and restoring original data based on common bit data and coordinate annotation information of the matching template.

[0021] Preferably, the device further comprises the following implementations:

[0022] The hardware acceleration chip pipelines compression and decompression operations; the template table and the data table adopt SRAM or DRAM or register storage structures, and the matching unit adopts a TCAM hardware acceleration module; and the address mapping unit and the direct mapping unit achieve nanosecond-level data access delay through pipeline design.

[0023] Preferably, the device further comprises the following implementations:

[0024] The hardware acceleration chip supports multi-core parallel processing, and improves the throughput of compression and decompression by dynamically allocating computing resources.

[0025] In a sixth aspect, the application provides an electronic device, comprising: a processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the processor, can implement the method of the first aspect.

[0026] In a seventh aspect, the application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to run to implement the method of the first aspect.

[0027] Compared with the prior art, the application has obvious advantages and beneficial effects, and at least has the following advantages:

[0028] 1) The mixed structure of templates and compressed data is adopted, which significantly improves the lossless compression efficiency, realizes entropy compression, and maintains high-speed data read-write performance.

[0029] 2) The coordinate annotation and common bit data recorded by the template are used to realize efficient compression and decompression, adapt to chip-level hardware design, and maximize the throughput performance in combination with a pipeline architecture.

[0030] 3) Based on the difference bit and the public bit position mark, the data is accurately restored, supports nanosecond-level data extraction and restoration, and ensures ultra-low delay compression and decompression.

[0031] 4) An on-the-way incremental automatic training mechanism is adopted to dynamically optimize the template, realize non-sensitization adaptive compression, and realize zero intervention of users throughout the process.

[0032] 5) The application scenarios are wide, and cover multi-field high-performance data compression requirements. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a preferred embodiment of the table entry compression method of the present application;

[0034] Figure 2 A data structure diagram of a preferred embodiment of the table entry compression method of the present application;

[0035] Figure 3 A data structure diagram of a preferred embodiment of the template of the present application;

[0036] Figure 4 A flowchart of a preferred embodiment of the table entry writing table of the present application;

[0037] Figure 5 A flowchart of a preferred embodiment of the table entry reading table of the present application;

[0038] Figure 6 A structure diagram of a preferred embodiment of the address mapping of the present application;

[0039] Figure 7 A structure diagram of a preferred embodiment of the direct mapping of the present application;

[0040] Figure 8 A flowchart of a preferred embodiment of the template training of the present application;

[0041] Figure 9 A schematic diagram of a preferred embodiment of the table entry compression device of the present application; DETAILED DESCRIPTION

[0042] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0043] Figure 1Fig. 1 is a flow diagram of a fast entry compression according to an exemplary embodiment. It should be noted that the implementation method of the embodiment of the present application can be applied to the implementation device of the embodiment of the present application, and the implementation device can be configured on an electronic device or in a server, wherein the electronic device can be a PC or a mobile terminal. The embodiment of the present application does not make too many limitations on this.

[0044] The entry compression in the present application adopts a data structure combining templates and compressed data, as shown in Figure 1 The implementation method of the entry compression includes the following steps:

[0045] In step S110, in response to a write table request of a to-be-stored entry, a matched template of the to-be-stored entry is determined, so as to determine whether to perform entry compression according to a matching state of the to-be-stored entry.

[0046] In step S120, in the case of performing entry compression, bit data of a corresponding position is extracted based on a difference bit of coordinate label information according to the matched template, and the compressed data is generated by splicing in an original position order, and the compressed data is stored in a data table.

[0047] In step S130, in the case of not performing entry compression, the original entry is stored in the data table.

[0048] For example, in order to clearly understand the present application, first, the basic structure of the entry compression is introduced in Figure 2 The entry compression includes two parts of templates and data in the figure. For example, the character B is taken as an entry 1, the original entry (data) is 01000010, the template A is 01000***, and the template B is 01001***. First, the original data is matched with the template A and the template B, the matching method is that the * position is ignored, and the rest of the positions are accurately matched, and it is found that the template A is matched and hit. Then, the data of the * position is extracted and spliced, so as to generate the compressed data 010. Finally, the template A is stored in the template table, and the compressed data is stored in the data table. It should be noted that the * in the template does not constrain its position, and the adjacent and irregular interval can be used. Secondly, if the original data and the template A and the template B are all unmatched, the original data is stored in the compressed data table.

[0049] Since the compressed data and the template are in a one-to-many relationship, the bit width of the compressed data is greatly reduced compared with the bit width of the original data. It can be seen that the present application can greatly compress the data by combining the template and the data, effectively utilize the storage resources, store the uncompressed data in full amount, and improve the adaptability and compatibility of the method.

[0050] In a preferred embodiment, as Figure 3As shown, the data structure of the template includes common bit data and coordinate annotation information. The template 01000*** can be divided into common bit data 01000 and coordinate annotation information 11111000, wherein 1 in the coordinate annotation information represents a common bit identifier, 0 represents a difference bit identifier, the common bit data 01000 can be known according to 1 in the coordinate annotation information, and the compressed data 010 can be extracted according to 0 in the coordinate annotation information. When the template is matched, the positions of all 1s in the coordinate annotation information are accurately matched, so as to determine the matched template; when the table entry is compressed, the positions of all 0s in the coordinate annotation information in the matched template are extracted and spliced, and the spliced data is the compressed data; when the table entry is decompressed, the positions of all 1s in the coordinate annotation information in the matched template are sequentially filled with the common bit data in the template, and the positions of all 0s in the coordinate annotation information in the matched template are sequentially filled with the compressed data in the data table, and the filled data is the original data.

[0051] Figure 4 FIG. 1 is a schematic diagram of a write table flow of ultrafast table entry compression according to an exemplary embodiment. The write table flow is divided into a write table flow of compressed table entries and a write table flow of non-compressible table entries.

[0052] The write table flow of compressible table entries includes the following steps:

[0053] Step S410, input data is obtained;

[0054] Step S420, the input data is matched with all templates in the template table as original data, and the bit data at the positions of all 1s in the coordinate annotation information in each template in the template table is different at least one.

[0055] Step S430, compressed data is extracted, and the bit data at the positions of all 0s in the coordinate annotation information in the template is extracted and spliced;

[0056] Step S440, the mapping table is updated, and the data is indexed with the matched template, which is used to determine whether it is compressed data, so as to query the compressed data and the template when decompressing;

[0057] Step S450, the compressed data is written into the data table;

[0058] Step S460, the template is written into the template table.

[0059] The write table flow of non-compressible table entries includes the following steps:

[0060] Step S410, input data is obtained;

[0061] Step S420, the input data is matched with all templates in the template table as original data, and the bit data at the positions of all 1s in the coordinate annotation information in each template in the template table is different at least one.

[0062] Step S470: Update the mapping table and mark it as incompressible data;

[0063] Step S480: Write all the original data into the data table.

[0064] Figure 5 This is a schematic diagram of a table reading process for ultra-fast table entry compression according to an exemplary embodiment. The table reading process is divided into a table reading process for compressed entries and a table reading process for incompressible entries.

[0065] The process of writing compressible table entries includes the following steps:

[0066] Step S510: Read the table entry based on the lookup index and determine it to be a compressed table entry;

[0067] Step S520: Obtain compressed data and a mapping table based on the table entry content. The mapping table data includes metadata such as template index.

[0068] Step S530: Read the template table according to the template index to obtain the corresponding template;

[0069] Step S540: Restore the table entries according to the template and compressed data. Fill all positions of 1 in the coordinate annotation information of the matching template with the common bit data in the template in sequence. Fill all positions of 0 in the coordinate annotation information of the matching template with the compressed data in the data table in sequence. The filled data is the original data.

[0070] Step S560: Output the original table entry data.

[0071] The process of writing incompressible table entries includes the following steps:

[0072] Step S510: Read the table entry based on the lookup index and determine that it is an incompressible table entry;

[0073] Step S550: Read the next data segment according to the mapping relationship in the mapping table until all data segments have been read.

[0074] Step S560: Output the original table entry data.

[0075] In a preferred embodiment, the mapping table is implemented as an address mapping scheme. For example... Figure 6 As shown, the table entry types include data tables, mapping tables, and template tables. Data tables are used to store compressed table entries (data); template tables are used to store templates corresponding to compressed data, facilitating data restoration; mapping tables are used to store table entry metadata, including data table indexes, compression tags, and template table indexes. Figure 6The original table entry of the compressed data is 1101101011, and the table entry index is 0. The matching template of the example table entry is 11*11***1*, which is stored in the template table (index 0). The template table stores data 110111101100010, and the common bit data in the template is the first 5 bits (11011). The last 10 bits are coordinate annotation information 1101100010, 1 indicating common data marking, and 0 indicating difference data marking. The compressed data of the example table entry is 00101, which is stored in the data table (index 0). The mapping table index is the same as the table entry index, the data table index in the mapping table is 0, the compression flag is 1, and the template table index is 0. When reading the compressed table entry, first read the mapping table according to the table entry index, then read the data and template according to the data table index and the template table index, and finally start data restoration based on the template according to the compression flag. When performing data restoration, the coordinate annotation information in the template 0 is 1101100010, the common bit data is 11011, and the compressed data is 00101. According to the 1 and 0 in the coordinate annotation information, the common bit data and the compressed data are filled respectively, and finally the original data 1101101011 is restored. Figure 6 The original table entry of the uncompressed data is 1010010111, and the table entry index is 3. The data index in the mapping table is 3, the compression flag is 0, and the template table index is 0 (which can be any value). The original table entry is stored in the data table, occupying two adjacent rows, in this example, table entry 3 and table entry 4. When reading the uncompressed table entry, first read the mapping table according to the table entry index, then judge according to the compression flag that it is uncompressed data, and finally read the two rows of the data table according to the data table index to obtain the original table entry.

[0076] In a preferred embodiment, the mapping table is implemented as a direct mapping scheme. As shown in Figure 7 As shown, the table entry type includes a data table and a template table. The data table is used to save the compressed table entry (data) and table entry metadata, including the compression flag, the next segment data index, and the template index; the template table is used to save the template corresponding to the compressed data, facilitating data restoration. Figure 7The original table entry of the compressed data is 1101101011, and the table entry index is 0. The matching template of the example table entry is 11*11***1* saved in the template table (index 0), and the template table stores data 110111101100010. The common bit data in the template is the first 5 bits (11011), and the last 10 bits are coordinate marking information (1101100010). 1 represents the common data mark, and 0 represents the difference data mark. The compressed data of the example table entry is 00101, the compression mark is 1, the next segment index is 0 (which can be any value), and the template table index is 0, which is saved in the data table (index 10). When reading the compressed table entry, first read the data table according to the table entry index, then read the template according to the template table index, and finally start data restoration based on the template according to the compression mark. When performing data restoration, the coordinate marking information in the template 0 is 1101100010, the common bit data is 11011, and the compressed data is 00101. According to the 1 and 0 in the coordinate standard information, the common bit data and the compressed data are filled respectively, and finally the original data 1101101011 is restored. Figure 7 The original table entry of the uncompressed data is 1010010111, and the table entry index is 3. The full amount of data is stored in multiple rows in the data table (in this example, table entry 3 and table entry 8), and the first data segment is stored in the table entry index table. The remaining data segments can be any table entry in the data table. When reading the uncompressed table entry, first read the data table according to the table entry index, then determine that it is uncompressed data according to the compression mark, read the next segment data according to the next segment index, and finally splice the read multiple segments to restore the original data.

[0077] Figure 8 A template training process diagram of ultra-fast table entry compression is shown according to an example embodiment. The template training includes the following steps:

[0078] Step S810, based on a clustering algorithm such as a greedy algorithm or a local sensitive hashing algorithm or an unsupervised clustering algorithm, the table entries are clustered and divided according to the Hamming distance.

[0079] Step S820, training each cluster: first set any table entry as the initial template, and initialize the common bit data of the template as the table entry data and the coordinate marking information as the full common bit data.

[0080] Step S830, reference count statistics: calculate the position of the different bits in the template and the first table entry in the cluster, and add one to the position. Repeat the above operation until all table entries in the cluster are traversed.

[0081] Step S840, based on the reference count, the template is expanded, the basic operation is to expand 1 bit to the bit position with the highest reference count, and the number of bits expanded each time can be configured;

[0082] Step S850, the expansion is repeated until the amount of public bit data of the template reaches the set value. After the template is trained for the current cluster, the same template training is performed for the next cluster until all clusters are trained.

[0083] In summary, the present application has the following advantages compared with the prior art:

[0084] 1) The mixed structure of template and compressed data fusion is adopted, which significantly improves the lossless compression efficiency, realizes entropy compression and maintains high-speed data reading performance.

[0085] 2) Through the coordinate marking and public bit data recorded by the template, efficient compression and decompression are realized, which is suitable for chip-level hardware design and can maximize the throughput performance combined with pipeline architecture.

[0086] 3) Based on the difference bit and public bit state marking, the data is accurately restored, supporting nanosecond-level data extraction and restoration, ensuring ultra-low delay compression and decompression.

[0087] 4) The on-the-way incremental automatic training mechanism is adopted to dynamically optimize the template, realize adaptive compression without induction, and realize zero user intervention throughout.

[0088] 5) The application scenarios are wide, covering high-performance data compression needs in multiple fields. It is not only suitable for in-chip table item data compression (such as CPU cache, GPU memory, DPU flow table, etc.), but also can efficiently support various high real-time data compression scenarios (such as weight compression in AI inference, KV optimization, scientific data compression in HPC, etc.).

[0089] Based on the same technical concept, the present application also provides a system and device for ultra-fast compression, such as Figure 9As shown, the compression device includes a table entry compression module 10, a table entry decompression module 20 and a template training module 30. The table entry compression module 10 includes a template matching unit 110, a compressed data extraction unit 120, a template table 130 and a data table 140. The template matching unit 10 is configured to match a write table entry with all templates in the template table 130 in response to a write table request of a table entry, and send the matching result and the matched template to the compressed data extraction unit 120. The compressed data extraction unit extracts compressed data based on the template and the original data, generates a data table index, compressed data (or full amount data) and metadata after the extraction is completed, and sends them to the data table 140. The data table 140 is used to store the compressed data (or full amount data) and the metadata. The table entry decompression module 20 includes a read compressed data and template unit 210 and a decompression unit 220. The read compressed data and template unit 210 is configured to read the template table 130 and the data table 140 according to a template index and a table entry index respectively in response to a read table request of a table entry, and send the template and the data to the decompression unit 220 to perform specific decompression operations. The template training module 30 includes a table entry data unit 310, a training unit 320, a standby template table 330 and a template replacement unit 340. The table entry data unit 310 is used to accept all original table entries and form training data. The training unit 320 clusters and trains the table entries in the table entry data 310, generates new templates after the training is completed, and saves the newly generated templates to the standby template table 330. The template replacement unit 340 replaces the templates according to the hot and cold of the templates of the template table 130 and the standby template table 330.

[0090] The specific embodiment details and more advantages of the ultrafast table entry compression according to the embodiments of the present application can be referred to the above-mentioned embodiments of the processing method of the ultrafast table entry compression, and will not be described here in more detail.

[0091] Correspondingly, the present application also provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to make the computer execute the write table method of the ultrafast table entry compression and the read table method of the ultrafast table entry compression according to the above-mentioned embodiments.

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0093] The units described in the embodiments of the present invention can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0094] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0095] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0096] The above description merely illustrates the preferred embodiment of the application and a principle of the applied technologies. It is understood by those skilled in the art that the disclosed range of the present application is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present application (but not limited to) can be formed.

[0097] Further, although operations are depicted in a particular, chronological sequence, this should not be understood as requiring the operations to be performed in that order, or to be performed sequentially. In some circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are contained in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0098] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for implementing ultra-fast term compression, characterized in that, The table entry compression includes a template and data, and the implementation method includes the following steps: The entries in the original data table are compared using binary bit patterns to quantify the bit-level similarity distribution characteristics between entries, thereby generating a template. The template includes common bit data and coordinate annotation information, wherein the coordinate annotation information determines the positions of common and differing bit bits using a binary mask or coordinate index. In response to a write request for a table entry to be stored, a matching template for the table entry to be stored is determined, so as to determine whether to perform table entry compression based on the matching status of the table entry to be stored; When performing table entry compression, according to the matching template, the corresponding bit data is extracted based on the position of the difference bit bits in the coordinate annotation information, and the compressed data is generated by concatenating them in the original position order. The compressed data is stored in the data table. Store the original table entries into the data table without performing table entry compression; The compressed data or original table entries are stored in a designated storage location within the data table, and a mapping table is used to record the indexes of the compressed data or original table entries. The mapping table employs an address mapping scheme or a direct mapping scheme to associate the original table entry index with the indexes of the data table and / or template table, enabling fast addressing during table entry compression and decompression. In response to a read request for a table entry, the differential data and template of the table entry to be read are obtained based on the data index and / or template index recorded in the mapping table. The position of the differential data in the original table entry is determined based on the differential bits of the coordinate annotation information of the template. The position of the common data in the original table entry is determined based on the common bits of the coordinate annotation information of the template. The table entry to be read is determined by concatenating the differential bit data position, the common bit data position, the compressed data in the data table, and the common bit data of the coordinate annotation information.

2. The implementation method according to claim 1, characterized in that, The method for generating the template includes: Based on the administrator's previous experience, users define and write templates into the template table; The template is automatically trained based on a similarity search algorithm. In the initial stage of the automatic training, the spare template set is left empty or filled with experience templates based on the administrator's previous experience. The common bits in the coordinate annotation information of the experience template are empty or not full. The matching status of each item in the training set in the spare template set is determined according to each item in the training set. The common bits and common data in the coordinate annotation information are expanded in the matching template. Compare the reference counts of the template sets and replace the templates with low reference counts with new templates, which are high-referenced templates from the spare template table.

3. The implementation method according to claim 2, characterized in that, The template includes: In response to a template replacement request, the table entries referencing the template to be replaced are traversed in the mapping table. If there is free space in the data table, the original table entries are restored according to the template to be replaced and the corresponding compressed data, and the original table entries are stored as new table entries in the data table. If the reference count of the replaced template in the template matching table reaches zero, the replaced template is deleted, and the highly referenced template in the standby template table is added to the template matching table. The traversal of table entries referencing the replaced template in the ejection table includes both actively triggering a data table scan and passively triggering the reading of table entries.

4. The implementation method according to claim 1, characterized in that, The template matching includes: In response to a storage request for a table entry, the table entry data is compared bit by bit with the template in the template table; If the data of the common bits are inconsistent, the template to be matched is replaced with the next one in the matching template table; if all are consistent, the template match is successful, and the template is the matching template. If the template matches, the data corresponding to the bit positions of the entry to be stored is extracted based on the difference bits in the matching template, and then sequentially concatenated to generate the compressed data, which is then saved to the data table. If the template does not match, the entire entry to be stored is stored in the data table.

5. The implementation method according to claim 1, characterized in that, The decompression of the table entries includes: The mapping table and / or data table are read based on the table entry index. The mapping table is used to determine whether the data is compressed, the template index corresponding to the compressed data, and the data index; the data table is used to store compressed data or raw data. If the data is uncompressed, the data table is read in a chain-like segmented manner and then concatenated to generate the original data. If the data is compressed, the compressed data and the matching template are obtained based on the data index and the template index. The data source of the bit position is determined according to the coordinate annotation information. The common bit data comes from the template, and the different bit data comes from the data table.

6. The implementation method according to claim 1, characterized in that, The address mapping scheme includes: Construct an independent mapping table to record the relationships between the original table entries or compressed data indexes, template indexes, and compressed tags; Decompression addressing is accelerated by reading the mapping table and template table in parallel.

7. The implementation method according to claim 1, characterized in that, The direct mapping scheme includes: The data table and the mapping table are merged, and the table entries of the mapping table include compression tags, template indexes and data segments; Incompressible data is linked to a reserved segment and an extended segment via a linked list pointer. The reserved segment stores the first segment of data, and the extended segment is dynamically allocated.

8. A high-speed meter compression system, characterized in that, Includes the following modules: Template training module: Supports manual template generation and automatic template training, including data storage unit, training algorithm execution unit, spare template storage unit, and template replacement execution unit. The table entry compression module includes a template matching unit, a compressed data extraction unit, a data table, and a template table, and is used to perform template matching, compressed data extraction, and data storage; the template matching unit includes a template table to be matched and a template matching processing unit. The data table stores compressed data that matches the template and raw data that does not match the template; the template table stores the templates corresponding to the data that matches the template. The table entry decompression module includes a compressed data and template reading unit and a decompression unit. The compressed data and template reading unit responds to a table entry read request by reading compressed data from the data table according to the data index and reading the matching template of the compressed data according to the template index. The decompression unit restores the table entry to be read based on the common bit data and coordinate annotation information of the matching template and returns it.

9. The system according to claim 8, characterized in that, The automatic training function of the template training module is implemented through hardware circuitry, including: Bit-level statistical circuits based on greedy algorithms, locality-sensitive hashing algorithms, and unsupervised clustering algorithms; Parallelized processing unit for incremental training and template replacement.

10. A high-speed meter compression device, characterized in that, include: A hardware acceleration chip, integrating the modules of the system described in claim 8, supports pipelined compression and decompression operations; The template table and data table use SRAM, DRAM or register storage structure, and the matching unit uses TCAM hardware acceleration module; The address mapping unit and the direct mapping unit achieve nanosecond-level data access latency through pipelined design.

11. The apparatus according to claim 10, characterized in that, The hardware acceleration chip supports multi-core parallel processing and improves the throughput of compression and decompression by dynamically allocating computing resources.

12. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images