Data compression methods, apparatus and systems

Abstract

This application provides a data compression method, apparatus, and system. The data compression method is applied to a programmable logic device and includes: acquiring input data; extracting characters to be processed from the input data to generate a target string; performing a calculation on the target string according to a preset hash function to determine hash data; reading multiple string addresses corresponding to the hash data based on a preset hash dictionary; reading multiple string data corresponding to the multiple string addresses, and matching the multiple string data with the target string; and compressing the characters to be processed based on the matching results. The data compression method, apparatus, and system provided in this application add a preset hash function calculation and a preset hash dictionary, and achieve fast compression of input data through programmable logic device hardware, thereby improving real-time compression capabilities.

Description

Technical Field

[0001] This application relates to the field of data compression technology, specifically to a data compression method, apparatus, and system. Background Technology

[0002] In industrial production and scientific research, signals are often sampled at high speed for extended periods, generating large amounts of sampled data. In certain special environments, limitations in size and power consumption restrict the addition of excessive memory, necessitating the introduction of data compression techniques. Software compression algorithms are computationally intensive, requiring high CPU speeds and large data caches, and are generally used in non-real-time compression applications where time requirements are not critical. Real-time data compression is typically implemented in hardware. Lossy compression results in data reconstruction that differs from the original. Most data acquisition systems, due to the uncertainty of the measured object, require lossless data compression. Lossless data compression is currently mostly implemented using dedicated compression chips, such as those from ADI (Advanced Digital Infrastructure).

[0003] In the process of conceiving and implementing this application, the inventors discovered at least the following problems: using a dedicated compression chip to achieve lossless data compression not only increases the difficulty and cost of hardware design, such as increasing the PCB (printed circuit board) design area and power consumption, and making heat dissipation of the circuit board more difficult, but also makes future system program upgrades more difficult and inconvenient to upgrade and maintain the program.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] To alleviate the above problems, this application provides a data compression method, apparatus, and system.

[0006] In one aspect, this application provides a data compression method, specifically applied to a programmable logic device, the data compression method comprising:

[0007] Acquire input data, extract the characters to be processed from the input data to generate a target string;

[0008] The target string is processed according to a preset hash function to determine the hash data;

[0009] Read multiple string addresses corresponding to the hash data based on a preset hash dictionary;

[0010] Read the corresponding string data based on the multiple string addresses, and match the target string based on the multiple string data;

[0011] The characters to be processed are compressed based on the matching results.

[0012] Optionally, the data compression method includes the following steps in performing the steps of acquiring input data and extracting the characters to be processed from the input data to generate the target string:

[0013] Obtain multi-byte input data and divide the input data into multiple single-byte data;

[0014] Extract each character to be processed and read the string prefix. Combine the string prefix and the character to be processed to form the target string.

[0015] Optionally, the data compression method includes the following steps in performing the extraction of characters to be processed one by one, reading string prefixes, and combining the string prefixes and the characters to be processed into the target string:

[0016] Read a preset number of single-byte data based on the plurality of single-byte data;

[0017] The preset number of single-byte data are output to the corresponding processing channels, so that multiple processing channels process the preset number of single-byte data in parallel.

[0018] Optionally, the preset hash function is a modulo function; the data compression method includes the following steps in performing the operation on the target string according to the preset hash function to determine the hash data:

[0019] The target string is moduloed by the preset prime number, and the remainder is used as the hash data.

[0020] Optionally, the data compression method includes the following steps in performing the step of reading multiple string addresses corresponding to the hash data based on a preset hash dictionary:

[0021] Input the hash data as the target hash address of the preset hash dictionary, and read the data stored at the target hash address;

[0022] Extract a preset number of string addresses from the data stored at the target hash address.

[0023] Optionally, the data compression method includes the following steps in performing the steps of reading corresponding string data based on the multiple string addresses and matching the target string based on the multiple string data:

[0024] Based on multiple string dictionaries, read the multiple string data stored corresponding to the multiple string addresses respectively;

[0025] The target string is compared with each of the plurality of string data.

[0026] Optionally, the data compression method includes the following steps in performing the comparison of the target string with the plurality of string data:

[0027] The string dictionary is constructed using a multi-port memory and a preset dictionary depth to perform parallel comparison processing on the multiple string data.

[0028] Optionally, the data compression method includes the following steps in performing the compression process on the characters to be processed based on the matching result:

[0029] When the target string is matched, no encoded value is output; and / or,

[0030] If the target string is not matched, the string prefix is ​​output as the encoding value, the target string is added to the string dictionary, and the storage address corresponding to the target string is added to the preset hash dictionary.

[0031] Optionally, after performing the step of compressing the character to be processed based on the matching result, the data compression method further includes:

[0032] When the target string is matched, the target string is used as a prefix to process the next character to be processed; and / or,

[0033] If the target string is not matched, the character to be processed is used as the prefix of the string to process the next character to be processed.

[0034] Optionally, the data compression method includes the following steps after performing the step of compressing the character to be processed based on the matching result:

[0035] A pseudo-encryption algorithm is used to encrypt the encoded values ​​in the maintained string dictionary.

[0036] On the other hand, this application also provides a data compression device, specifically, the data compression device includes a processor and a memory;

[0037] The memory stores a computer program that, when executed by the processor, implements the steps of the data compression method described above; and / or,

[0038] The data compression device includes a string module, a hash algorithm module, a compression dictionary module, an encoding module, and an encryption module;

[0039] The data compression device is implemented using a programmable logic device;

[0040] The string module is used to acquire input data and extract the characters to be processed from the input data to generate a target string.

[0041] The hash algorithm module is used to perform calculations on the target string according to a preset hash function to determine the hash data;

[0042] The compression dictionary module is used to read multiple string addresses corresponding to the hash data based on a preset hash dictionary, read multiple string data corresponding to the multiple string addresses, and match the target string based on the multiple string data;

[0043] The encoding module is used to compress the characters to be processed based on the matching results;

[0044] The encryption module is used to encrypt the compressed characters to be processed using a pseudo-encryption algorithm.

[0045] Optionally, the compression dictionary module in the data compression device includes a hash dictionary unit, at least one string dictionary unit, and a matching decision unit;

[0046] The hash dictionary unit is used to update the preset hash dictionary and read multiple string addresses corresponding to the hash data based on the preset hash dictionary;

[0047] Each string dictionary unit is used to update the string dictionary and read the corresponding string data according to the multiple string addresses;

[0048] The matching decision unit is used to match the target string based on the plurality of string data.

[0049] Optionally, the string dictionary unit in the data compression device uses a multi-port memory to build the string dictionary, so that the matching decision unit can perform parallel comparison processing on the multiple string data.

[0050] On the other hand, this application also provides a data compression system, specifically, the data compression system includes a data distribution module, at least one data compression device and a data aggregation module;

[0051] The data compression system is implemented using programmable logic devices;

[0052] The data allocation module acquires multi-byte input data and divides the input data into multiple single-byte data.

[0053] Each data compression device processes a corresponding single byte of data in parallel and generates a compression result for each.

[0054] The data aggregation module merges the compression results generated by each data compression device and outputs the total compressed data.

[0055] As described above, the data compression method, apparatus and system provided in this application, based on the principle of string list compression algorithm, add preset hash function operation and preset hash dictionary, and combine the high integration, low power consumption, flexibility and parallel operation characteristics of programmable logic device, to realize fast compression of input data through programmable logic device hardware, thereby improving real-time compression capability. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0057] Figure 1 This is a flowchart of a data compression method according to an embodiment of this application.

[0058] Figure 2 This is a structural diagram of a dual-port memory according to an embodiment of this application.

[0059] Figure 3 This is a timing diagram illustrating a data compression method implemented by a data compression apparatus according to an embodiment of this application.

[0060] Figure 4 This is a structural diagram of a data compression apparatus according to an embodiment of this application.

[0061] Figure 5 This is a structural diagram of a compressed dictionary module according to an embodiment of this application.

[0062] Figure 6 This is a structural diagram of a data compression system according to an embodiment of this application.

[0063] The realization of the objectives, functional features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0064] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0066] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0067] It should be noted that the data compression method, apparatus, and system provided in this application are implemented based on the principle of the string table compression algorithm. The string table compression algorithm (LZW compression algorithm, Lempel-Ziv-Welch Encoding) is a compression method invented by Abraham Lempel, Jacob Ziv, and Terry Welch that uses a table lookup algorithm to compress files into smaller files. Two commonly used file formats for the string table compression algorithm are GIF image format and TIFF image format, which are used for websites. The string table compression algorithm is also suitable for compressing text files.

[0068] String list compression algorithms achieve lossless data compression by creating a string table and using shorter codes to represent longer strings. As a lossless compression algorithm, string list compression has adaptive characteristics, maintaining good compression performance even when the statistical characteristics of the signal are unclear. It extracts different characters from the original text file data, creates a compilation table based on these characters, and then replaces the corresponding characters in the original text file data with the indices of the characters in the compilation table, reducing the original data size. This compilation table is not pre-created but dynamically created based on the original file data; during decoding, the original compilation table must be reconstructed from the encoded data.

[0069] First Embodiment

[0070] On one hand, this application provides a data compression method. Figure 1 This is a flowchart of a data compression method according to an embodiment of this application.

[0071] Please see Figure 1 In one embodiment, the data compression method is applied to a programmable logic device.

[0072] For example, the programmable logic device can be a field-programmable gate array (FPGA). FPGAs have advantages such as high clock frequency, low internal latency, pure hardware parallel control, fast operation speed, flexible programming configuration, short development cycle, strong anti-interference capability, and abundant internal resources, making them very suitable for real-time high-speed data compression. Optionally, this application does not limit the type of programmable logic device, and FPGA can be selected.

[0073] Data compression methods include:

[0074] S10: Obtain input data, extract the characters to be processed from the input data, and generate the target string.

[0075] By converting input data into different target strings, a string table can be easily created, enabling lossless data compression by representing longer strings with shorter code. For example, on an FPGA, generating a target string from a single byte of input data (the character to be processed) requires one clock cycle.

[0076] S20: Perform calculations on the target string according to the preset hash function to determine the hash data.

[0077] Using hash data allows for a quick way to check if a target string exists in a character dictionary. For example, on an FPGA, a preset hash function performs a hash operation on the target string to generate hash data, which requires one clock cycle.

[0078] S30: Read multiple string addresses corresponding to hash data based on a preset hash dictionary.

[0079] Based on the principle of hash collisions, multiple string addresses may correspond to the same hash data. Therefore, multiple string addresses can be stored as a list of addresses corresponding to the same hash data.

[0080] S40: Read multiple string data corresponding to multiple string addresses, and match the target string based on the multiple string data.

[0081] Among the multiple string addresses read, only one string corresponds to the target string. By matching and verifying each string individually, the target string can be accurately determined. For example, on an FPGA, a preset hash dictionary reads multiple string addresses corresponding to hash data, reads multiple string data corresponding to these addresses, and matches the target string against these string data, requiring three clock cycles.

[0082] S50: Compress the characters to be processed based on the matching results.

[0083] When a matching string is found for the target string, the encoding corresponding to the target string can be determined and compressed. For example, on an FPGA, compressing the character to be processed based on the matching result requires one clock cycle. Optionally, encrypting the compressed character to be processed requires one clock cycle. Understandably, compressing a single byte of data requires seven clock cycles. If the operating clock frequency (CLK) of the data compression method is 200MHz and the clock cycle is 5ns, then compressing one byte of data takes 35ns (27.25MB / s). Increasing the clock frequency of the data compression method can improve the compression rate; if the clock frequency (CLK) is 300MHz, the compression rate of the data compression method is 40.07MB / s. Optionally, this application does not limit the clock cycle required for each step from S10 to S50.

[0084] In this embodiment, the data compression method, based on the principle of string list compression algorithm, adds preset hash function operation and preset hash dictionary. Combining the high integration, low power consumption, flexibility and parallel operation characteristics of programmable logic device, the input data is compressed quickly through programmable logic device hardware, thereby improving real-time compression capability.

[0085] In one embodiment, the data compression method includes the following steps in S10: acquiring input data and extracting the characters to be processed from the input data to generate the target string:

[0086] S11: Obtain multi-byte input data and divide the input data into multiple single-byte data.

[0087] For example, 4 bytes of input data are received and split into 4 single-byte data to facilitate processing for each single-byte data.

[0088] S12: Extract each character to be processed and read the string prefix, then combine the string prefix and the characters to be processed into the target string.

[0089] For example, multiple single-byte data are extracted one by one into characters to be processed. Each character to be processed includes a single byte of data. The target string is then formed by combining each single byte of data with the first digit of the string prefix. For example, the single byte of data, i.e., the character to be processed, occupies 8 bits, and the string prefix, which is the address of the character to be processed or the string stored in the string dictionary, occupies 11 bits. Therefore, the target string consists of the character to be processed and the string prefix, occupying a total of 19 bits. Allocating a fixed-length 11-bit storage space in the FPGA to store the string address can prevent variable-length strings from requesting the maximum length of 19 bits of storage space, thus avoiding wasting FPGA storage resources.

[0090] In one embodiment, the data compression method includes the following steps in S12: extracting characters to be processed one by one and reading string prefixes, combining string prefixes and characters to be processed into a target string:

[0091] S13: Read a preset number of single-byte data based on multiple single-byte data;

[0092] S14: Output the preset number of single-byte data to the corresponding processing channels, so that multiple processing channels can process the preset number of single-byte data in parallel.

[0093] For example, the hash dictionary module receives hash data and outputs a list of addresses in the string dictionary where the current string String = P+C is stored. There are 12 addresses in total, and each of the 12 addresses stores 12 different strings String = P+C. However, the hash values ​​corresponding to these 12 different strings String = P+C are the same. The first string dictionary module, the second string dictionary module, and the third string dictionary module each receive 4 different addresses, and then each string dictionary module reads the strings corresponding to the 4 addresses from the string dictionary.

[0094] In one embodiment, the preset hash function is the modulo function.

[0095] Optionally, this application does not limit the type of the preset hash function. A suitable preset hash function type is selected by comprehensively considering its operation speed, efficiency, and the uniqueness of the hash value. For example, the preset hash function can be a modulo function. Since the modulo function is relatively fast, only one modulo operation is needed to complete the hash operation, thus ensuring the compression rate. Using an FPGA to implement the modulo operation, only one clock cycle is needed to complete the hash operation.

[0096] The data compression method, in executing S20: the step of determining the hash data by performing calculations on the target string according to a preset hash function, includes:

[0097] S21: Perform a modulo operation on the target string with a preset prime number, and use the remainder as the hash data.

[0098] For example, the target string is received and hashed using a preset hash function to generate a second-digit hash data, thereby quickly checking whether the target string exists in the string dictionary. The hash data and the target string are equivalent, and the preset hash function can also be reversed to restore the target string from the hash data. The modulo function formula is: h(x) = x mod m, where h(x) is the hash data, x is the target string, and m is a preset prime number. Optionally, this application does not limit the size of m; m can be any constant. However, when choosing the value of m, it is best to choose a prime number to avoid too many keywords mapping to one position, thus reducing the efficiency of the modulo function. For example, converting a 19-bit target string into 11-bit binary hash data results in hash data containing 2^11 = 2048 bytes, and m can be a prime number close to 2048, such as 2039.

[0099] In one embodiment, the data compression method includes the following steps in performing S30: reading multiple string addresses corresponding to hash data based on a preset hash dictionary:

[0100] S31: Input hash data as the target hash address of the preset hash dictionary, and read the data stored at the target hash address;

[0101] S32: Extract a preset number of string addresses from the data stored at the target hash address.

[0102] For example, a preset hash dictionary is used for fast matching of target strings. Since the hash data generated by the preset hash function is not unique, hash collisions can occur, meaning different strings using the preset hash function will generate the same hash data. Therefore, to resolve hash collisions, the preset hash dictionary stores multiple different string addresses corresponding to the same hash data. Understandably, for each target hash address received, the preset hash dictionary can read a preset number of addresses of different strings in the string dictionary. The preset hash dictionary is implemented using the FPGA's internal storage resource, RAM (Random Access Memory).

[0103] For example, according to the principle of string list compression algorithm, a string dictionary needs to be built during the string list compression process to store strings. The depth of the dictionary determines the string list compression ratio and the string lookup speed. A shallower (fewer) or deeper (more) dictionary depth will result in a lower string list compression ratio. Similarly, the depth and width of the preset hash dictionary for storing multiple string addresses corresponding to hash addresses must also be considered. This application does not limit the depth of the preset hash dictionary. If the first 19-bit string is hashed to generate the second 11-bit binary hash data, the preset hash dictionary depth can be chosen to be 2048, and the hash address width can be 11 bits. This application does not limit the data width stored at the hash address of the preset hash dictionary. The data width stored at the hash address represents a preset number of multiple string addresses, and an appropriate width can be selected based on the preset hash dictionary depth and the number of hash collisions. If this application is based on video grayscale image compression, the hash collision situation can be counted using software. For a string with a depth of 2K, approximately 11 hash collisions will occur. The preset hash dictionary can choose to store 12 different string addresses for each hash address.

[0104] For example, please refer to Tables 1 and 2. Table 1 is a hash dictionary table of one embodiment, and Table 2 is a hash address data table based on the embodiment in Table 1. In Table 1, RAM Address represents the hash address of hash data, with an address width of 11 bits and a dictionary depth of 2048. RAM Data represents the data stored at the hash address, with a width of 132 bits. Since the data bit width is 132 binary bits, and the address bit width of a string in the string dictionary is 11 bits, the data stored at the hash address can store the addresses corresponding to 12 different strings. In Table 2, the same hash data address can store 12 different string addresses, namely the corresponding 12 addresses Ad1 to Ad12.

[0105]

[0106]

[0107] Table 1: Hash Dictionary Table

[0108]

[0109] Table 2: Hash Dictionary Data Format Table

[0110] In one embodiment, the data compression method includes the following steps in S40: reading multiple string data corresponding to multiple string addresses and matching a target string based on the multiple string data:

[0111] S41: Based on multiple string dictionaries, read multiple string data stored at the addresses of multiple strings respectively;

[0112] S42: Compare the target string with multiple string data respectively.

[0113] For example, the string dictionary is implemented using the FPGA's internal RAM storage resources, enabling string matching functionality. This application does not limit the depth and width of the string dictionary. If the first 19-bit string is stored in an 11-bit string address, the string dictionary's depth is 2048, the string address width is 11 bits, and the string width is 19 bits. The string dictionary reads multiple string data points from multiple string addresses and matches the target string with each of these string data points to determine if a unique string matches the target string.

[0114] Please refer to Table 3, which is a string dictionary table for one embodiment. In Table 3, RAM Address represents the string address, with an address width of 11 bits and a dictionary depth of 2048. RAM Data represents the binary data corresponding to the string address, with a dictionary width of 19 bits.

[0115] RAM Address (11 bits) RAM Data (19 bits) 0 Data0 1 Data1 2 Data2 …… …… 2045 Data2045 2046 Data2046 2047 Data2047

[0116] Table 3: String Dictionary

[0117] In one embodiment, the data compression method includes the following step in performing S42: comparing the target string with multiple string data respectively:

[0118] S43: Use a multi-port memory and a preset dictionary depth to build a string dictionary for parallel comparison of multiple string data.

[0119] For example, string dictionaries can increase the speed of string searching and matching by performing parallel comparison of multiple string data through multi-port storage.

[0120] Figure 2 This is a structural diagram of a dual-port memory according to an embodiment of this application.

[0121] Please see Figure 2 Taking a dual-port RAM as an example to build a string dictionary, one port checks if the string at the first address is equal to the target string, and the other port checks if the string at the second address is equal to the target string. Thus, in one clock cycle, it can check if the strings at two addresses are equal to the target string, and in two clock cycles, it can check if the strings at four addresses are equal to the target string.

[0122] For example, since a preset hash dictionary receives multiple different addresses at once (the strings stored in the string dictionary at these multiple different addresses can all be equal to the target string), multiple string dictionaries can be created for target string matching. Optionally, this application does not limit the number of string dictionaries; a suitable number of string dictionaries can be selected based on the creation cost and search speed. For example, taking three string dictionaries as an example, when dynamically creating multiple string dictionaries, the stored content is the same, that is, the stored content of the three string dictionaries is identical. Searching for four different strings in one string dictionary requires two clock cycles. Utilizing the parallel processing capability of the FPGA, three string dictionaries can simultaneously search for 12 different strings, and each string dictionary searching for four different strings only requires two clock cycles, thus achieving fast string matching. If searching for 12 different addresses requires one clock cycle, then six string dictionaries need to be searched simultaneously.

[0123] For example, using three string dictionaries, please refer to Tables 4, 5, and 6. Table 4 shows the matching results of string dictionary A in one embodiment, Table 5 shows the matching results of string dictionary B in another embodiment, and Table 6 shows the matching results of string dictionary C in yet another embodiment. In the tables, `compare` indicates the comparison type, `String` is the target string, `String1` to `x` represent the strings 1 to `x` that need to be matched, and `Match1` to `3` represent the matching results of string dictionaries A to C, respectively. We can set `4'b0001` to indicate a successful match, and all other matching results to indicate unsuccessful matches. In practice, taking 12 different string addresses as an example, string dictionary A searches for the matching results of `String` with `String1` to `4`, string dictionary B searches for the matching results of `String` with `String5` to `8`, and string dictionary C searches for the matching results of `String` with `String9` to `12`. If a match is successful, there is exactly one matching result of `4'b0001`, which means that `String` and `String1` are matched successfully. If a match is unsuccessful, then none of the matching results will be `4'b0001`.

[0124] Compare Match1 String = String1 4‘b0001 String = String2 4‘b0010 String = String3 4‘b0100 String = String4 4‘b1000 Other 4‘b0000

[0125] Table 4: Matching Results of String Dictionary A

[0126] Compare Match2 String = String1 4‘b0001 String = String2 4‘b0010 String = String3 4‘b0100 String = String4 4‘b1000 Other 4‘b0000

[0127] Table 5: Matching Results of String Dictionary B

[0128] Compare Match3 String = String1 4‘b0001 String = String2 4‘b0010 String = String3 4‘b0100 String = String4 4‘b1000 Other 4‘b0000

[0129] Table 6: Matching Results of String Dictionary C

[0130] For example, the search speed and compression ratio of the target string differ in string dictionaries with different depths. A string dictionary with sufficient capacity will not fill up quickly, and due to the large number of stored strings, the data matching effect is good, effectively improving the compression ratio. An excessively large string dictionary increases the dictionary lookup time, affecting the compression speed, and also consumes a large amount of storage space. Optionally, this application does not limit the preset dictionary depth; a suitable preset dictionary depth can be determined through software algorithm experiments on a computer. The size of the dictionary designed in the software experiment must be within the acceptable range of the internal block RAM resources of the hardware FPGA chip, while also considering the hardware implementation speed of the algorithm.

[0131] For example, an infrared video segment is captured by an infrared camera and stored in an infrared storage hard drive. Then, the infrared video data is exported from the solid-state drive on a PC, and this segment of infrared data is used as the data source for compression. Software is used to compress the raw infrared data using a data compression algorithm within string dictionaries at different dictionary depths, and an optimal dictionary size or depth is selected as the preset dictionary depth.

[0132] Please refer to Table 7, which shows the data compression results of one embodiment. Dictionary depths of 512, 1024, 2048, and 4096 were set respectively. Table 7 shows that the compression ratios are best with dictionary depths of 2048 and 4096. Further selection was made based on FPGA internal RAM resources and string search speed. Since FPGA internal RAM resources are limited, a dictionary depth of 2048 occupies less RAM resources, so this depth was chosen. Additionally, a smaller dictionary depth results in faster string searches, so a dictionary depth of 2048 was selected. The original data was 388800*8bit, and data compression was performed on 388800 single-byte data. The compression rate describes the effect of file compression; it is the ratio of the compressed file size to the original size. For example, compressing a 100MB file to 90MB results in a compression rate of 90 / 100*100% = 90%. Generally, a lower compression rate is better. Furthermore, the larger the input data is compressed, the shorter the decompression time.

[0133]

[0134] Table 7: Data Compression Results

[0135] In the above embodiments, establishing a preset hash dictionary based on a preset hash function to quickly find the string address corresponding to the target string can improve the search speed and compression rate of the target string. The string dictionary receives multiple string addresses and reads out the corresponding strings to match the target string one by one.

[0136] In one embodiment, the data compression method includes the following steps in performing S50: compressing the characters to be processed according to the matching result:

[0137] S51: When the target string is matched, the encoded value is not output;

[0138] S52: When a target string is matched, use the target string as a prefix to process the next character to be processed.

[0139] For example, if the target string is in the string dictionary, the encoding value Code is not output temporarily. The string prefix is ​​expanded to the character to be processed plus the string prefix, i.e., the target string, to process the next character to be processed. If there is no next character to be processed, the string prefix that has been expanded to the target string is output as the encoding value.

[0140] In one embodiment, the data compression method includes the following steps in performing S50: compressing the characters to be processed according to the matching result:

[0141] S53: When no target string is matched, output the string prefix as the encoding value, add the target string to the string dictionary, and add the storage address corresponding to the target string to the preset hash dictionary.

[0142] For example, the string dictionary can implement string update functionality and dynamically maintain the string dictionary, while the preset hash dictionary can also dynamically maintain hash addresses. If the target string is not in the string dictionary, the target string is added to the string dictionary, and the address of the target string is added to the corresponding hash address in the preset hash dictionary.

[0143] S54: If the target string is not matched, use the character to be processed as a prefix to process the next character to be processed.

[0144] For example, if the target string does not match successfully, the string prefix is ​​replaced with the character to be processed in order to process the next character to be processed. If there is no next character to be processed, the string prefix that has been replaced with the character to be processed is output as the encoded value.

[0145] In one embodiment, the data compression method includes the following steps after performing S50: compressing the character to be processed according to the matching result:

[0146] S55: Use a pseudo-encryption algorithm to encrypt the encoded values ​​in the maintained string dictionary.

[0147] For example, a pseudo-encryption algorithm is used to encrypt the compressed encoded value Code. Table 8 is a pseudo-encryption algorithm table for one embodiment. Referring to Table 8, the binary data of Code before encryption is 000111001000, and the binary data of En_code after encryption is 00010011100. Encrypting data is to protect data from being captured by criminals. Optionally, this application does not limit the encryption method of the encoded value. Different encryption algorithms implemented on the FPGA will require different clock cycles. A suitable encryption method can be selected according to the compression rate requirements. If a pseudo-encryption algorithm is selected, data encryption can be completed in one clock cycle by exchanging bits in the output.

[0148] Code 11th 10 people 9 people 8-bit 7 people 6 people 5 people 4 people 3 people 2 people 1 person En_Code 1 person 2 people 3 people 4 people 5 people 10 people 9 people 8-bit 7 people 6 people 11th Code 0 0 1 1 1 0 0 1 0 0 0 En_Code 0 0 0 1 0 0 1 1 1 0 0

[0149] Table 8: Pseudo-encryption algorithm table

[0150] Second Embodiment

[0151] On the other hand, this application also provides a data compression device.

[0152] In one embodiment, the data compression apparatus includes a processor and a memory. The memory stores a computer program that, when executed by the processor, implements the steps of the data compression method described above.

[0153] Figure 3 This is a timing diagram illustrating a data compression method implemented by a data compression apparatus according to an embodiment of this application.

[0154] Please see Figure 3 In one embodiment, the data compression device inputs the character to be processed, a single-byte data C. It combines this single-byte data with a string prefix P to generate the target string P+C, requiring one clock cycle. A preset hash function calculates the hash data of the target string, requiring one clock cycle. A preset hash dictionary and a string dictionary search for a matching target string based on the hash data, requiring three clock cycles. The device outputs the encoded value Code based on the matching result Match, requiring one clock cycle. The device receives the encoded value, encrypts it, and outputs the encrypted data Encode, requiring one clock cycle. Then, the next character to be processed, C, is input.

[0155] Figure 4 This is a structural diagram of a data compression apparatus according to an embodiment of this application.

[0156] Please see Figure 4 In one embodiment, the data compression device includes a string module 10, a hash algorithm module 20, a compression dictionary module 30, an encoding module 40, and an encryption module 50.

[0157] The data compression device is implemented using a programmable logic device.

[0158] For example, a programmable logic device can be an FPGA.

[0159] The string module 10 is used to acquire input data, extract the characters to be processed from the input data to generate the target string, and the input data is single-byte data. The hash algorithm module 20 is used to perform calculations on the target string according to a preset hash function to determine the hash data. The compression dictionary module 30 is used to read multiple string addresses corresponding to the hash data based on the preset hash dictionary, read multiple string data corresponding to the multiple string addresses, and match the multiple string data with the target string. The encoding module 40 is used to compress the characters to be processed according to the matching results. The encryption module 50 is used to encrypt the compressed characters to be processed using a pseudo-encryption algorithm.

[0160] In this embodiment, the data compression device adds a hash algorithm module 20 and a compression dictionary module 30 to the string table compression algorithm principle. Combining the high integration, low power consumption, flexibility and parallel computing characteristics of programmable logic devices, the device achieves fast compression of single-byte input data through programmable logic device hardware, thereby improving real-time compression capabilities.

[0161] Figure 5 This is a structural diagram of a compressed dictionary module according to an embodiment of this application.

[0162] Please see Figure 5 In one embodiment, the compression dictionary module 30 in the data compression device includes a hash dictionary unit 31, at least one string dictionary unit 32, and a matching decision unit 33.

[0163] Hash dictionary unit 31 is used to update the preset hash dictionary and read multiple string addresses corresponding to the hash data based on the preset hash dictionary. Each string dictionary unit 32 is used to update the string dictionary and read multiple string data corresponding to the multiple string addresses. Matching decision unit 33 is used to match the target string based on the multiple string data.

[0164] For example, each string dictionary unit 32 contains a string dictionary with identical content. Optionally, this application does not limit the number of string dictionary units 32, and an appropriate number of string dictionary units 32 can be selected according to the construction cost and search speed. Figure 5 The text shows three string dictionary units 32.

[0165] In this embodiment, the data compression device can quickly search whether the target string exists in the string dictionary by combining the hash dictionary unit 31 with multiple string dictionary units 32, thereby enabling the matching decision unit 33 to quickly output the matching result.

[0166] In one embodiment, the string dictionary unit 32 in the data compression device uses a multi-port memory to build a string dictionary so that the matching decision unit 33 can perform parallel comparison processing on multiple string data.

[0167] For example, each string dictionary unit 32 can be based on a multi-port memory to simultaneously and in parallel compare string data of the corresponding number of ports.

[0168] Third Embodiment

[0169] On the other hand, this application also provides a data compression system. Figure 6 This is a structural diagram of a data compression system according to an embodiment of this application.

[0170] Please see Figure 6 In one embodiment, the data compression system includes a data distribution module 1, at least one data compression device 2, and a data aggregation module 3.

[0171] Data compression systems are implemented using programmable logic devices.

[0172] For example, a programmable logic device can be an FPGA.

[0173] Data allocation module 1 acquires multi-byte input data and divides it into multiple single-byte data. Each data compression device 2 processes its corresponding single-byte data, thus processing multiple single-byte data in parallel and generating a compression result for each. Data aggregation module 3 merges the compression results generated by each data compression device 2 and outputs the total compressed data.

[0174] For example, parallel compression of multiple single-byte data units 2 by multiple data compression devices 2 can improve the data compression rate. If the operating clock of the data compression device 2 is 200MHz and the clock period is 5ns, and the compression rate of one data compression device 2 is 27MB / s, then the compression rate of four data compression devices 2 is 108MB / s. Optionally, this application does not limit the number of data compression devices 2; the more data compression devices 2 there are, the more single bytes can be compressed in parallel each time. A suitable number of data compression devices 2 is selected by comprehensively considering the compression rate and cost. Figure 6 Four data compression devices 2 are shown in the figure.

[0175] In this embodiment, the data compression system utilizes the high integration, low power consumption, flexibility, and parallel computing characteristics of the data compression device 2 combined with the programmable logic device to achieve rapid compression of multi-byte input data through programmable logic device hardware, thereby improving real-time compression capabilities.

[0176] As described above, the data compression method, apparatus, and system provide a lossless compression method with real-time compression, compression stability, and ease of software maintenance and upgrades, overcoming the technical bottlenecks and hardware limitations imposed by dedicated compression chips. During the compression process, all data processing and transmission are handled by the FPGA. Considering the real-time compression speed, the dictionary cache utilizes the FPGA's internal RAM resources to address the slow compression rate of existing FPGA-implemented LZW schemes. Dictionary creation is accomplished using FPGA on-chip resources. While a large-capacity dictionary improves the compression ratio, the available FPGA resources must be considered. Using a fixed-length encoding method (11 bits) for data processing solves the dictionary creation and update issues and addresses the limited dictionary capacity (less than 1024) of existing FPGA-implemented LZW schemes, resulting in a low LZW compression rate. Converting the 11-bit data into an 8-bit data stream for transmission and storage solves the problem of transmitting and storing the compressed output data stream.

[0177] It should be noted that step designations such as S10 and S20 are used in this application for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S20 first and then S10, etc., but these should all be within the protection scope of this application.

[0178] In the embodiments of the data compression apparatus and system provided in this application, all the technical features of any of the above-described method embodiments may be included. The extended and explanatory content of the specification is basically the same as that of the various embodiments of the above methods, and will not be repeated here.

[0179] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform the methods described in the various possible implementations above.

[0180] This application also provides a chip, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device with the chip installed performs the methods described in the various possible implementations above.

[0181] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0182] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0183] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0184] The units in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0185] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0186] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0187] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0188] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data compression method characterized by, The data compression method is applied to a programmable logic device, and comprises the following steps: Obtaining input data, extracting a to-be-processed character of the input data to generate a target string; Operating the target string according to a preset hash function to determine hash data; Inputting the hash data as a target hash address of a preset hash dictionary, reading data stored in the target hash address, and extracting a preset number of string addresses from the data stored in the target hash address; Reading a plurality of string data corresponding to the plurality of string addresses according to the plurality of string addresses, and matching the target string according to the plurality of string data; According to the matching result, the to-be-processed character is compressed.

2. The data compression method of claim 1, wherein, The step of obtaining input data and extracting a to-be-processed character of the input data to generate a target string comprises the following steps: Obtaining multi-byte input data, and dividing the input data into a plurality of single-byte data; Extracting the to-be-processed character one by one and reading a string prefix, and combining the string prefix and the to-be-processed character to generate the target string.

3. The data compression method of claim 2, wherein, The step of extracting the to-be-processed character one by one and reading a string prefix, and combining the string prefix and the to-be-processed character to generate the target string comprises the following steps: Reading a preset number of single-byte data based on the plurality of single-byte data; Outputting the preset number of single-byte data to corresponding processing channels respectively, so that a plurality of processing channels process the preset number of single-byte data in parallel.

4. The data compression method of claim 1, wherein, The preset hash function is a modulo function; the step of operating the target string according to a preset hash function to determine hash data comprises the following steps: Taking the target string modulo a preset prime number, and taking the remainder as the hash data.

5. The data compression method according to any one of claims 1 to 4, characterized in that, The step of reading a plurality of string data corresponding to the plurality of string addresses according to the plurality of string addresses, and matching the target string according to the plurality of string data comprises the following steps: Reading a plurality of string data corresponding to the plurality of string addresses based on a plurality of string dictionaries respectively; Comparing the target string with the plurality of string data respectively.

6. The data compression method of claim 5, wherein, The step of comparing the target string with the plurality of string data respectively comprises the following steps: Using a multi-port memory and a preset dictionary depth to build the string dictionary to perform parallel comparison processing on the plurality of string data.

7. The data compression method of claim 5, wherein, The step of compressing the to-be-processed character according to the matching result comprises the following steps: When the target string is matched, no encoding value is outputted; and / or, When the target string is not matched, the string prefix is outputted as an encoding value, the target string is added to a string dictionary, and a storage address corresponding to the target string is added to the preset hash dictionary.

8. The data compression method of claim 7, wherein, The step of compressing the to-be-processed character according to the matching result further comprises the following steps: When the target string is matched, the target string is taken as the string prefix to process the next to-be-processed character; and / or, When the target string is not matched, the to-be-processed character is taken as the string prefix to process the next to-be-processed character.

9. The data compression method of claim 7, wherein, The step of compressing the to-be-processed character according to the matching result comprises the following steps of: The encoding values in the completed string dictionary are encrypted by using a pseudo-encryption algorithm.

10. A data compression device, characterized by, The data compression device comprises a processor and a memory; the memory stores a computer program, and the computer program is executed by the processor to implement the steps of the data compression method according to any one of claims 1-9; And / or, the data compression device comprises a string module, a hash algorithm module, a compression dictionary module, an encoding module and an encryption module; The data compression device is implemented by using a programmable logic device; The string module is configured to obtain input data, extract to-be-processed characters of the input data to generate a target string; The hash algorithm module is configured to perform operation on the target string according to a preset hash function to determine hash data; The compression dictionary module is configured to read a plurality of string addresses corresponding to the hash data based on a preset hash dictionary, wherein the compression dictionary module receives the hash data input from the hash algorithm module as a target hash address of the preset hash dictionary, reads data stored in the target hash address, extracts a preset number of string addresses from the data stored in the target hash address, reads a plurality of string data corresponding to the plurality of string addresses according to the plurality of string addresses, and matches the target string according to the plurality of string data; The encoding module is configured to compress the to-be-processed character according to the matching result; The encryption module is configured to encrypt the to-be-processed character after compression by using a pseudo-encryption algorithm.

11. The data compression device of claim 10, wherein, The compression dictionary module comprises a hash dictionary unit, at least one string dictionary unit and a matching decision unit; The hash dictionary unit is configured to update the preset hash dictionary and read a plurality of string addresses corresponding to the hash data based on the preset hash dictionary; The string dictionary unit is configured to update a string dictionary and read a plurality of string data corresponding to the plurality of string addresses; The matching decision unit is configured to match the target string according to the plurality of string data.

12. The data compression device of claim 11, wherein, The string dictionary unit uses a multi-port memory to build the string dictionary, so that the matching decision unit performs parallel comparison processing on the plurality of string data.

13. A data compression system characterized by, The data compression system comprises a data distribution module, at least one data compression device according to claim 10 and a data aggregation module; The data compression system is implemented by using a programmable logic device; The data distribution module obtains multi-byte input data, and divides the input data into a plurality of single-byte data; Each data compression device processes a corresponding single-byte data to process the plurality of single-byte data in parallel and correspondingly generate a compression result; The data aggregation module combines the compression results generated by each data compression device and outputs total compression data.

Images