Data compression method, compressor, device, electronic equipment, chip and medium
By matching multiple first strings in parallel, using compressed characters in the compression cache, using LZ77 and Hoffman encoder, the high processor load problem caused by serial compression in the prior art is solved, and efficient and accurate data compression is achieved.
Patent Information
- Application Number
- CN202411087649.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2025-07-25
AI Technical Summary
Data compression algorithms in the prior art, such as UDC, are serially compressed, resulting in high processor load and inability to efficiently handle compression tasks of large data volumes.
By matching multiple first strings in parallel, using compressed characters in the compression cache to determine the target compressed data, and using LZ77 and Hoffman encoder to improve compression efficiency and accuracy.
Improves the efficiency and accuracy of data compression, reduces processor load, and ensures that data is compressed in character order.
Smart Images
Figure CN120377933A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data processing, and particularly to a data compression method, a compressor, a device, an electronic device, a chip, and a medium. Background Art
[0002] With the continuous development of current network technologies, the amount of data produced by humans is growing exponentially, and network-based applications are also increasing day by day. A large amount of data needs to be transmitted over the network, which poses higher requirements for data storage and transmission. It is necessary to store data under limited hardware resources and transmit as much data as possible under limited bandwidth. Therefore, it is very necessary to effectively compress data before data transmission and storage. Summary of the Invention
[0003] The present disclosure aims to at least solve one of the technical problems in the related art to some extent.
[0004] A first aspect embodiment of the present disclosure provides a data compression method, including:
[0005] Obtaining a plurality of first strings that meet a preset quantity from the data to be compressed;
[0006] Based on the compressed characters in the compression cache, respectively matching the plurality of first strings to obtain a first matching result for each of the first strings;
[0007] Determining target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed.
[0008] A second aspect embodiment of the present disclosure provides a compressor, including: a serial-parallel conversion module, a preset number of calculation modules, and a first encoder; wherein,
[0009] The serial-parallel conversion module is configured to obtain a plurality of first strings that meet a preset quantity from the data to be compressed;
[0010] The preset number of calculation modules are configured to, based on the compressed characters in the compression cache, respectively match the plurality of first strings to obtain a first matching result for each of the first strings;
[0011] The first encoder is configured to determine target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed.
[0012] A third aspect embodiment of the present disclosure provides a data compression device, including:
[0013] An acquisition module, configured to acquire a plurality of first strings meeting a preset quantity from data to be compressed;
[0014] A matching module, configured to respectively match the plurality of first strings based on the compressed characters in a compression cache, so as to obtain a first matching result of each first string;
[0015] A determination module, configured to determine target compressed data according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed.
[0016] An embodiment of the fourth aspect of the present disclosure provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the data compression method provided in the embodiment of the first aspect of the present disclosure is implemented.
[0017] An embodiment of the fifth aspect of the present disclosure provides a chip, where the chip includes a compressor, and the compressor is configured to execute the data compression method provided in the embodiment of the first aspect.
[0018] An embodiment of the sixth aspect of the present disclosure provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the data compression method provided in the embodiment of the first aspect of the present disclosure is implemented.
[0019] The data compression method, compressor, device, electronic device, chip, and medium provided by the present disclosure have the following beneficial effects:
[0020] In the embodiments of the present disclosure, a plurality of first strings meeting a preset quantity are acquired from data to be compressed; the plurality of first strings are respectively matched based on the compressed characters in a compression cache, so as to obtain a first matching result of each first string; and target compressed data is determined according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed. Thus, the plurality of first strings can be respectively and concurrently matched based on the compressed characters in the compression cache, the quantity of first strings processed simultaneously is increased, and further the length of strings compressed each time is increased, effectively improving the compression efficiency. Moreover, based on the positions of the first strings in the data to be compressed and the first matching results, the target compressed data is determined, so that while improving the compression efficiency, the data to be compressed can be compressed in character order, ensuring the compression accuracy.
[0021] Additional aspects and advantages of the present disclosure will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present disclosure. Description of the Drawings
[0022] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:
[0023] Figure 1 It is a schematic flowchart of a data compression method provided by an embodiment of the present disclosure;
[0024] Figure 2 It is a schematic flowchart of a data compression method provided by another embodiment of the present disclosure;
[0025] Figure 3 It is a schematic diagram of determining a first matching result provided by an embodiment of the present disclosure;
[0026] Figure 4 It is a schematic diagram of a first-in first-out provided by an embodiment of the present disclosure;
[0027] Figure 5 It is a schematic flowchart of a data compression method provided by an embodiment of the present disclosure;
[0028] Figure 6 It is a schematic structural diagram of a compressor provided by an embodiment of the present disclosure;
[0029] Figure 7 It is a schematic structural diagram of another compressor provided by an embodiment of the present disclosure;
[0030] Figure 8 It is a schematic structural diagram of a data compression device provided by an embodiment of the present disclosure;
[0031] Figure 9 It shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present disclosure;
[0032] Figure 10 It is a schematic structural diagram of a chip proposed by an embodiment of the present disclosure. Detailed Embodiments
[0033] The embodiments of the present disclosure are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present disclosure, but should not be construed as limiting the present disclosure.
[0034] The data compression method, compressor, device, electronic device, chip and medium of the embodiments of the present disclosure are described below with reference to the accompanying drawings.
[0035] In the related art, uplink data compression (UDC) is a compression algorithm based on the Deflate kernel, which is a classic text data compression algorithm. However, the UDC compression algorithm is a serial compression and decompression algorithm, and the compression of subsequent data depends on the prior information of the previous data. Therefore, it is necessary to compress byte by byte in order, which occupies a relatively high processor load.
[0036] The data compression method in the embodiments of the present disclosure can be applied to the field of wireless communication data encoding and compression, and can also be applied to the compressed storage of text files, the compressed transmission of hypertext protocol data such as the Hypertext Transfer Protocol (HTTP) and the Real-time Transport Protocol (RTP) in the communication field, etc.
[0037] Figure 1 It is a flowchart of a data compression method provided by an embodiment of the present disclosure.
[0038] In the embodiments of the present disclosure, the data compression method is configured in a data compression device as an example. The data compression device can be applied to any electronic device so that the electronic device can perform the data compression function.
[0039] As Figure 1 shown, the data compression method may include the following steps:
[0040] Step 101, obtain a plurality of first strings that meet a preset quantity from the data to be compressed.
[0041] Among them, the data to be compressed is the data to be compressed.
[0042] In some embodiments, the data to be compressed may be data that the compressor has previously read from the memory into the internal cache of the compressor and is to be compressed.
[0043] Among them, the preset quantity may be the number of threads that can perform the first string matching at the same time. In some embodiments, the number of threads can be determined according to the number of calculation modules in the compressor. For example, the preset quantity can be 4, 5, etc. The present disclosure does not limit this.
[0044] Among them, the first string may contain a second quantity of characters. The second quantity may be the number of characters processed by each thread each time set in advance. For example, the second quantity can be 3, 4, etc. The present disclosure does not limit this.
[0045] In some embodiments, the number of characters processed by each thread each time may be the same or different. The present disclosure does not limit this.
[0046] In some embodiments, the first number of characters may be read starting from the first character of the data to be compressed to obtain a fifth string, and then the fifth string is processed to obtain a plurality of first strings that meet the preset number.
[0047] Among them, the first number can be determined according to the number of parallel threads and the number of characters processed by each thread each time.
[0048] For example, if the number of parallel threads is 4 and the number of characters processed by each thread each time is 3, then the first number is 12. That is, 12 characters are read starting from the first character of the data to be compressed as the fifth string.
[0049] In some embodiments, when the number of characters processed by each thread each time is the same, the fifth string can be evenly divided into a preset number of first strings.
[0050] For example, the fifth string is "Iamastudenty", the number of parallel threads is 4, and each thread processes three characters, then four first strings are obtained, namely "Iam", "ast", "ude", "nty".
[0051] In some embodiments, when the number of characters processed by each thread each time is different, the fifth string can be divided according to the number of characters processed by each thread to obtain a preset number of first strings.
[0052] For example, the first string is "Iamastudenty", the number of characters processed by the first thread each time is 3, the number of characters processed by the second thread each time is 4, and the number of characters processed by the third thread each time is 5. Then the three first strings obtained after dividing the fifth string are "Iam", "astu", "denty".
[0053] In some embodiments, if the data to be compressed is stored in the internal cache of the compressor, the number of characters of the data to be compressed stored in the internal cache should be greater than or equal to the quantity threshold. In some embodiments, the quantity threshold = the length of the first string * the preset number + (258 - the length of the first string).
[0054] Step 102, based on the compressed characters in the compression cache, match each of the plurality of first strings to obtain a first matching result for each first string.
[0055] Among them, the compressed characters in the compression cache can be the characters that have been successfully compressed and stored in the compression cache before compressing the data to be compressed. For example, before compressing the data to be compressed, if "giafbsidmdehifbg" has been compressed, then "giafbsidmdehifbg" is stored in the compression cache.
[0056] For example, there are three first strings, namely "Iam", "ast", and "ude". The first thread matches "Iam" in the data to be compressed with the compressed characters in the compression cache; the second thread matches "ast" in the data to be compressed with the compressed characters in the compression cache; the third thread matches "ude" in the data to be compressed with the compressed characters in the compression cache.
[0057] In some embodiments, the first matching result may include: the target offset position of the compressed character that matches the first string successfully in the compression cache; the first length corresponding to the consecutive identical characters between the characters starting from the first string in the data to be compressed and the characters starting from the target offset position in the compression cache.
[0058] In some embodiments, when the first string matches the compressed characters in the compression cache successfully, the characters after the first string in the data to be compressed can be continuously matched with the subsequent characters of the string that matches the first string successfully in the compression cache until the first character does not match and then stop. The first length of the successfully matched characters and the target offset position of the first successfully matched character in the compression cache are determined as the first matching result.
[0059] For example, the first string is "Iam", the subsequent characters in the compression cache that match "Iam" successfully are "aboy", the characters after "Iam" in the data to be compressed are "astudenty", and the first non-identical character is "s". Then the first length is determined to be 4, and the offset position of the first character "I" in the compression cache is the target offset position.
[0060] For example, the first string is "nty", the subsequent characters in the compression cache that match "nty" successfully are "abcfg", the characters after "nty" in the data to be compressed are "abcde", and the first non-identical character is "d". Then the first length is determined to be 6, and the offset position of the first character "n" in the compression cache is the target offset position.
[0061] In some embodiments, if a first string corresponds to multiple matching results, the matching result with the longest first length is used as the first matching result corresponding to the first string.
[0062] For example, the first string is "nty", the target offset of the first matching result is 100, the length is 5, the target offset position of the second matching result is 200, and the length is 6. Then, the target offset position in the first matching result corresponding to "nty" is 200, and the first length is 6.
[0063] In some embodiments, if the data to be compressed is stored in the internal cache of the compressor, when matching the first string and the subsequent characters in the data to be compressed with the compressed characters in the compression cache, the characters can be directly read from the internal cache of the compressor, improving the reading efficiency and thus the matching efficiency.
[0064] Step 103: Determine the target compressed data according to the obtained multiple first matching results and the positions of each first string in the data to be compressed.
[0065] In some embodiments, an LZ77 (Lempel-Ziv 1977) encoder can be used to encode the multiple first matching results and the fifth string to obtain the target compressed data. Thus, by using the LZ77 encoding algorithm and encoding the fifth string based on the multiple first matching results, the storage space of the data can be effectively reduced.
[0066] It should be noted that some first strings can match the same string in the compression cache, while some first strings may not match the same string in the compression cache, and the corresponding first matching results can be empty.
[0067] In some embodiments, if the strings corresponding to the first lengths in each first matching result do not overlap, the substrings in the fifth string corresponding to the first matching results can be directly replaced based on the first matching results to obtain the target compressed data.
[0068] For example, the fifth string is "Iamastudenty", the first strings are "Iam", "ast", "ude", "nty" respectively. Among them, the first length in the first matching result corresponding to "Iam" is 3, and the target offset position is 2000; the first length in the first matching result corresponding to "ast" is empty; the first length in the first matching result corresponding to "ude" is empty; the first length in the first matching result corresponding to "nty" is 6, and the target offset position is 1000. Then the corresponding target compressed data is "(2000, 3)astude(1000, 6)".
[0069] For example, the fifth string is "Iamastudenty", and the first strings are "Iam", "ast", "ude", "nty" respectively. Among them, the first length in the first matching result corresponding to "Iam" is 3, and the target offset position is 2000. The first length in the first matching result corresponding to "ast" is empty. The first length in the first matching result corresponding to "ude" is empty. The first length in the first matching result corresponding to "nty" is 3, and the target offset position is 1000. Then the corresponding target compressed data is "(2000, 3)astude(1000, 3)".
[0070] In some embodiments, when the strings corresponding to the first lengths in the first matching results of one first string and the strings corresponding to the first lengths in the first matching results of another first string partially overlap, it is necessary to remove the duplicates from the overlapping part, and then obtain the target compressed data. This can avoid the characters in the overlapping part from being repeatedly compressed.
[0071] In some embodiments, a Huffman encoder can be used to compress the target compressed data based on the Huffman code table.
[0072] In the embodiments of the present disclosure, a plurality of first strings meeting a preset quantity are obtained from the data to be compressed; the plurality of first strings are respectively matched based on the compressed characters in the compression cache to obtain the first matching results of each first string; the target compressed data is determined according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed. Thus, the plurality of first strings can be parallelly matched respectively based on the compressed characters in the compression cache, increasing the quantity of the first strings processed simultaneously, and further increasing the length of the strings compressed each time, effectively improving the compression efficiency. And determining the target compressed data based on the positions of the first strings in the data to be compressed and the first matching results can ensure that the data to be compressed is compressed in character order while improving the compression efficiency, ensuring the compression accuracy.
[0073] Figure 2 It is a schematic flowchart of a data compression method provided by an embodiment of the present disclosure, as Figure 2 shown. The data compression method may include the following steps:
[0074] Step 201, obtain a plurality of first strings meeting a preset quantity from the data to be compressed.
[0075] Among them, for the specific implementation form of step 201, reference can be made to the detailed descriptions in other embodiments of the present disclosure, and details will not be elaborated here.
[0076] Step 202, determine the target hash value corresponding to the first string.
[0077] In some embodiments, each thread calculates the hash value corresponding to each first string respectively based on a preset hash function.
[0078] Figure 3 A schematic diagram of determining a first matching result provided by an embodiment of the present disclosure is as Figure 3 shown. The fifth string is "Iamastudenty", the first strings are "Iam", "ast", "ude", "nty" respectively, the hash value corresponding to "Iam" is 1, the hash value corresponding to "ast" is x, the hash value corresponding to "ude" is y, and the hash value corresponding to "nty" is z.
[0079] Step 203: Query the hash table based on the target hash value to obtain the target offset position of the target hash value in the compression cache, where the hash table contains the offset positions of the second strings corresponding to the hash values in the compression cache.
[0080] Wherein, the hash table (Hash Table) is generated according to the hash values corresponding to the characters in the compression cache.
[0081] As Figure 3 shown, the offset position of the hash value 1 in the hash table is 2043, that is, the hash value corresponding to "Iam" starting from position 2043 in the compression cache is 1, the offset position of the hash value x in the hash table is 2035, that is, the hash value corresponding to three characters starting from position 2035 in the compression cache is x, the offset position of the hash value y in the hash table is 2039, that is, the hash value corresponding to three characters "boy" starting from position 2039 in the compression cache is y, and the offset position of the hash value z in the hash table is 2041, that is, the hash value corresponding to three characters "mab" starting from position 2041 in the compression cache is z.
[0082] Step 204: Match the characters starting from the first string in the data to be compressed with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters.
[0083] As Figure 3 shown, match the characters starting from "Iam" in the data to be compressed with the characters starting from position 2043 in the compression cache, and determine that the continuously identical characters are "Iama", then the first length is 4. Match the characters starting from "ast" in the data to be compressed with the characters starting from position 2035 in the compression cache, and determine that the first length is 0, that is, the first matching result is empty. Similarly, determine that the first matching results corresponding to "nty" and "Iam" are both empty.
[0084] It should be noted that the hash values calculated for different strings may be the same. Therefore, there may be a situation where the second string in the compression cache has the same hash value as the first string but does not match. In the hash table, one hash value can also correspond to one or more offset positions.
[0085] In some embodiments, if there are multiple offset positions for the target hash value corresponding to the first string in the hash table, then the characters starting from the first string in the data to be compressed are matched with the characters starting from each offset position in the compression cache to obtain the matching result corresponding to each offset position, and the matching result with the longest length is determined as the first matching result corresponding to the first string.
[0086] Step 205: Determine the target compressed data according to the obtained multiple first matching results and the position of each first string in the data to be compressed.
[0087] As Figure 3 shown, the target compressed data is "(2043,4)student".
[0088] Step 206: Determine the third string in the data to be compressed corresponding to the target compressed data.
[0089] For example, if the data to be compressed is "Iamastudenty...", and the target compressed data is "(2043,4)student", then the third string is "Iamastudenty".
[0090] For example, if the data to be compressed is "Iamastudentyabcfg...", and the target compressed data is "(2000,4)stude(1000,6)", then the third string is "Iamastudentyabc".
[0091] Step 207: Update the characters in the compression cache based on the third string in a first-in, first-out manner.
[0092] In some embodiments, the first-in, first-out manner can be implemented using an array, that is, the third string is filled at the end of the array.
[0093] Figure 4 For a schematic diagram of the first-in, first-out provided in an embodiment of the present disclosure, as Figure 4 shown, the data from "2047 - 2036" in the compression cache is deleted, the data after 2035 is shifted left, and the third string "Iamastudenty" is filled in the position from "11 - 0" in the compression cache.
[0094] In some embodiments, the first-in-first-out (FIFO) manner can also be implemented by using a linked list, that is, deleting the data with the earliest storage time in the compression cache and filling the third string at the position of the deleted data.
[0095] Step 208: Update the hash table based on the offset position of the first string in the compression cache and the corresponding target hash value.
[0096] In some embodiments, if the FIFO manner of an array is adopted, the hash table needs to be updated according to the updated position of each data in the compression cache, the offset position of the first string in the compression cache, and the corresponding target hash value.
[0097] In some embodiments, if the FIFO manner of a linked list is adopted, only the hash table needs to be updated according to the offset position of the first string in the compression cache and the corresponding target hash value. Less data is updated and the update speed is faster.
[0098] In the embodiments of the present disclosure, updating the hash table can ensure the accuracy of the hash table in real time.
[0099] In the embodiments of the present disclosure, a plurality of first strings meeting a preset quantity are obtained from the data to be compressed, and the target hash value corresponding to the first string is determined. Based on the target hash value, the hash table is queried to obtain the target offset position of the target hash value in the compression cache. Then, the characters starting from the first string in the data to be compressed are matched with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters. Finally, based on the obtained multiple first matching results and the position of each first string in the data to be compressed, the target compressed data is determined. Thus, by querying the hash table through the target hash value corresponding to the first string, the target offset position is quickly found, and then the characters starting from the first string in the data to be compressed are matched with the characters starting from the target offset position in the compression cache, improving the matching efficiency and further improving the data compression efficiency.
[0100] Figure 5 It is a schematic flowchart of a data compression method provided by an embodiment of the present disclosure. As Figure 5 shown, the data compression method may include the following steps:
[0101] Step 501: Obtain a plurality of first strings meeting a preset quantity from the data to be compressed.
[0102] Step 503: Match each of the plurality of first strings based on the compressed characters in the compression cache to obtain the first matching result of each first string.
[0103] Among them, the specific implementation forms of steps 501 to 502 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be specifically elaborated here.
[0104] Step 503, determine a target matching result according to multiple first matching results.
[0105] In some embodiments, the first matching result with the largest first length and greater than the length threshold can be determined as the target matching result.
[0106] It should be noted that the first length being greater than the length threshold can reduce the computational amount of subsequent compression and improve the compression efficiency.
[0107] In some embodiments, the length threshold can be determined according to the number of characters included in the first string. For example, the length threshold can be equal to the second quantity, or equal to the difference between the second quantity and 1.
[0108] For example, the fifth string is "Iamastudenty", and the first strings are "Iam", "ast", "ude", "nty" respectively. Among them, the first length in the first matching result corresponding to "Iam" is 4, the first length in the first matching result corresponding to "ast" is 5, the first length in the first matching result corresponding to "ude" is 7, and the first length in the first matching result corresponding to "nty" is 3. Then the target matching result is the first matching result corresponding to "ude". If the target offset position corresponding to "ude" is 1500, the target matching result is "(1500, 6)".
[0109] In some embodiments, from multiple first matching results, determine second matching results with a first length greater than the length threshold. When the number of second matching results is multiple, determine the fourth string corresponding to each second matching result in the compression cache. When there is partial overlap between two fourth strings, deduplicate the fourth strings, and update the first length in the corresponding second matching result based on the length of the deduplicated fourth strings to obtain the target matching result.
[0110] For example, the fifth string is "Iamastudenty", and the first strings are "Iam", "ast", "ude", "nty". Among them, the first matching result corresponding to "Iam" has a first length of 9 and a target offset position of 100. The first matching result corresponding to "ast" is empty. The first length in the first matching result corresponding to "ude" is 10, and the target offset position is 500. The first length in the first matching result corresponding to "nty" is 3. The length threshold is 3. Then, there is partial overlap between the first matching result corresponding to "Iam" and the first matching result corresponding to "ude", and the overlap length is 3. Therefore, the first length in the first matching result corresponding to "Iam" can be updated to 6. Then, the target matching results are the updated matching result (100, 6) of "Iam" and the first matching result (500, 10) corresponding to "ude".
[0111] Step 504: Replace the first length of characters starting from the first string character corresponding to the target matching result in the data to be compressed with the target matching result to obtain the target compressed data.
[0112] For example, if the data to be compressed is "Iamastudentyabcfghijk...", the fifth string is "Iamastudenty", the first strings are "Iam", "ast", "ude", "nty", and the target matching result is the first matching result corresponding to "ude" which is "(1500, 6)", then the corresponding target compressed data is "(1500, 6)denty", and the remaining data to be compressed is "abcfghijk...".
[0113] For example, if the data to be compressed is "Iamastudentyabcfghijk...", the fifth string is "Iamastudenty", the first strings are "Iam", "ast", "ude", "nty", and the target matching results are the updated matching result (100, 6) of "Iam" and the first matching result (500, 10) corresponding to "ude", then the target compressed data is "(100, 6)(500, 10)", and the remaining data to be compressed is "ghijk...".
[0114] In the embodiments of the present disclosure, a plurality of first strings that meet a preset quantity are obtained from the data to be compressed. Then, based on the compressed characters in the compression cache, the plurality of first strings are respectively matched to obtain a first matching result for each first string. According to the plurality of first matching results, a target matching result is determined. Finally, the first length of characters starting from the first string corresponding to the target matching result in the data to be compressed is replaced with the target matching result to obtain the target compressed data. Thus, by processing the plurality of first matching results to obtain the target matching result, it is possible to avoid storing overlapping strings in two first matching results, which may cause repeated compression and reduce the compression efficiency, thereby ensuring the accuracy of compression.
[0115] In some embodiments, when compressing a complete Internet Protocol (IP) packet, after compressing a part of the characters each time, the compressed data is cached until all the characters in the IP packet are compressed, and then the complete compressed result is output byte by byte in the form of an uplink data compression (UDC) frame.
[0116] In some embodiments, before compression, the checksum of the compression cache can be calculated according to the information in the compression cache, and this checksum is added to the frame header of the UDC.
[0117] To implement the above embodiments, the present disclosure also proposes a compressor.
[0118] Figure 6 It is a schematic structural diagram of the compressor provided by the embodiments of the present disclosure. As Figure 6 shown, the data compression device 600 may include: a serial-to-parallel conversion module 601, a preset number of calculation modules 602, a first encoder 603, and a compression cache 604; wherein,
[0119] The serial-to-parallel conversion module 601 is configured to obtain a plurality of first strings that meet a preset quantity from the data to be compressed;
[0120] The preset number of calculation modules 602 are configured to respectively match the plurality of first strings based on the compressed characters in the compression cache to obtain a first matching result for each first string;
[0121] The first encoder 603 is configured to determine the target compressed data according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed.
[0122] In some embodiments, the serial-to-parallel conversion module 601 is configured to:
[0123] Obtain the first quantity of characters starting from the first character of the data to be compressed to obtain a fifth string;
[0124] Process the fifth string to obtain multiple first strings that meet a preset quantity.
[0125] In some embodiments, the first matching result includes:
[0126] The target offset position in the compression cache of the compressed characters that match the first string successfully;
[0127] The first length corresponding to the consecutive identical characters between the characters starting from the first string in the data to be compressed and the characters starting from the target offset position in the compression cache.
[0128] In some embodiments, the first encoder 603 is used for:
[0129] Determine a target matching result according to multiple first matching results;
[0130] Replace the first length of characters starting from the first string character corresponding to the target matching result in the data to be compressed with the target matching result to obtain target compressed data.
[0131] In some embodiments, the first encoder 603 is used for:
[0132] Determine the first matching result with the largest first length and greater than the length threshold as the target matching result.
[0133] In some embodiments, the first encoder 603 is used for:
[0134] Determine a second matching result with a first length greater than the length threshold from multiple first matching results; in the case where the number of second matching results is multiple, determine the fourth string corresponding to each second matching result in the compression cache;
[0135] Deduplicate the fourth strings in the case of partial overlap of two fourth strings;
[0136] Update the first length in the corresponding second matching result based on the deduplicated fourth strings to obtain the target matching result.
[0137] In some embodiments, the first encoder may be an LZ77 (Lempel-Ziv 1977) encoder. The present disclosure does not limit this.
[0138] In the compressor according to the embodiments of the present disclosure, multiple computing modules can be used to match multiple first strings in parallel based on the compressed characters in the compression cache, increasing the number of first strings processed simultaneously, thereby increasing the length of the string compressed each time, effectively improving the compression efficiency. Moreover, based on the position of the first string in the data to be compressed and the first matching result, the target compressed data can be determined, which can compress the data to be compressed in character order while improving the compression efficiency, ensuring the compression accuracy.
[0139] Figure 7 FIG. [FIG. number] is a schematic structural diagram of another compressor provided by the embodiments of the present disclosure. As Figure 7 shown, the computing module 602 includes a hash calculation unit 6021, a hash query unit 6022, and a matching unit 6023:
[0140] The hash calculation unit 6021 is configured to determine the target hash value corresponding to the first string;
[0141] The hash query unit 6022 is configured to query the hash table based on the target hash value to obtain the target offset position of the target hash value in the compression cache, where the hash table contains the offset positions of the second strings corresponding to each hash value in the compression cache;
[0142] The matching unit 6023 is configured to match the characters starting from the first string in the data to be compressed with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters.
[0143] In some embodiments, the computing module further includes a compression cache reading unit 6024, configured to read the characters starting from the target offset position from the compression cache.
[0144] As Figure 7 shown, the compressor further includes a compression cache update module and a hash update module.
[0145] The compression cache update module is configured to determine the third string corresponding to the target compressed data in the data to be compressed, and update the characters in the compression cache based on the third string in a first-in, first-out manner;
[0146] The hash update module is configured to update the hash table based on the offset position of the first string in the compression cache and the corresponding target hash value.
[0147] As Figure 7 shown, the compressor further includes a second encoder, configured to compress the target compressed data.
[0148] In some embodiments, the second encoder may be a Huffman encoder, which is used to compress the target compressed data based on a Huffman code table.
[0149] To implement the above embodiments, the present disclosure also proposes a data compression device.
[0150] Figure 8 It is a schematic structural diagram of the data compression device provided by the embodiments of the present disclosure.
[0151] As Figure 8 shown, the data compression device 800 may include:
[0152] An acquisition module 801, configured to acquire a plurality of first strings that meet a preset quantity from the data to be compressed;
[0153] A matching module 802, configured to match the plurality of first strings respectively based on the compressed characters in the compression cache to obtain a first matching result for each first string;
[0154] A determination module 803, configured to determine the target compressed data according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed.
[0155] In some embodiments, the first matching result includes:
[0156] The target offset position of the compressed character that matches the first string successfully in the compression cache;
[0157] The first length corresponding to the continuously identical characters between the characters starting from the first string in the data to be compressed and the characters starting from the target offset position in the compression cache.
[0158] In some embodiments, the matching module 802 is configured to:
[0159] Determine the target hash value corresponding to the first string;
[0160] Based on the target hash value, query the hash table to obtain the target offset position of the target hash value in the compression cache, where the hash table contains the offset positions of the second strings corresponding to the hash values in the compression cache;
[0161] Match the characters starting from the first string in the data to be compressed with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters.
[0162] In some embodiments, it further includes an update module, configured to:
[0163] Determine the third string in the data to be compressed corresponding to the target compressed data;
[0164] Update the characters in the compression cache in a first-in-first-out manner based on the third string;
[0165] Update the hash table based on the offset position of the first string in the compression cache and the corresponding target hash value.
[0166] In some embodiments, the determination module 803 is configured to:
[0167] Determine a target matching result according to multiple first matching results;
[0168] Replace the first length of characters starting from the first character string corresponding to the target matching result in the data to be compressed with the target matching result to obtain target compressed data.
[0169] In some embodiments, the determination module 803 is configured to:
[0170] Determine the first matching result with the largest first length and greater than the length threshold as the target matching result.
[0171] In some embodiments, the determination module 803 is configured to:
[0172] Determine a second matching result with a first length greater than the length threshold from multiple first matching results;
[0173] In the case where the number of second matching results is multiple, determine the fourth string corresponding to each second matching result in the compression cache;
[0174] In the case where there is partial overlap between two fourth strings, remove the duplicates from the fourth strings;
[0175] Update the first length in the corresponding second matching result based on the length of the deduplicated fourth string to obtain the target matching result.
[0176] In some embodiments, the acquisition module 801 is configured to:
[0177] Obtain the first number of characters starting from the first character of the data to be compressed to obtain a fifth string;
[0178] Process the fifth string to obtain multiple first strings that meet the preset quantity.
[0179] For the functions and specific implementation principles of the above modules in the embodiments of the present disclosure, reference may be made to the above method embodiments, and details are not described herein again.
[0180] In an embodiment of the present disclosure, a plurality of first strings that meet a preset quantity are obtained from data to be compressed; the plurality of first strings are respectively matched based on the compressed characters in the compression cache to obtain a first matching result for each first string; and target compressed data is determined according to the obtained plurality of first matching results and the positions of each first string in the data to be compressed. Accordingly, the plurality of first strings can be matched in parallel based on the compressed characters in the compression cache, increasing the quantity of first strings processed simultaneously, thereby increasing the length of the strings compressed each time, effectively improving the compression efficiency. Moreover, based on the positions of the first strings in the data to be compressed and the first matching results, the target compressed data is determined, which can ensure the compression accuracy while improving the compression efficiency and compressing the data to be compressed in character order.
[0181] To implement the above embodiment, the present disclosure further provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the data compression method as proposed in the foregoing embodiment of the present disclosure is implemented.
[0182] Figure 9 The block diagram of an exemplary electronic device suitable for implementing the embodiments of the present disclosure is shown. Figure 9 The illustrated electronic device 12 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.
[0183] As Figure 9 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0184] The bus 18 represents one or more of several types of bus architectures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus architectures. For example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnection (PCI) bus.
[0185] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0186] The memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 9 not shown, commonly referred to as a "hard disk drive"). Although Figure 9 not shown in, a disk drive for reading and writing on a removable non-volatile disk (such as a "floppy disk") can be provided, as well as an optical disk drive for reading and writing on a removable non-volatile optical disk (such as a compact disc read only memory (CD-ROM), a digital video disc read only memory (DVD-ROM) or other optical media). In these cases, each drive can be connected to the bus 18 through one or more data media interfaces. The memory 28 can include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present disclosure.
[0187] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 42 generally perform the functions and / or methods in the embodiments described in the present disclosure.
[0188] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0189] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.
[0190] To implement the foregoing embodiments, the present disclosure also proposes a chip, which includes a compressor configured to execute the data compression method provided in the foregoing embodiments.
[0191] Figure 10 is a schematic structural diagram of the chip proposed in the embodiments of the present disclosure. Reference can be made to Figure 10 the schematic structural diagram of the chip 1000 shown, but not limited thereto.
[0192] The chip 1000 includes a compressor 1001 configured to execute any of the above methods.
[0193] In some embodiments, the chip 1000 further includes one or more interface circuits 1002. Optionally, the interface circuit 1002 is connected to the memory 1003. The interface circuit 1002 can be used to receive signals from the memory 1003 or other devices, and the interface circuit 1002 can be used to send signals to the memory 1003 or other devices. For example, the interface circuit 1002 can read the instructions stored in the memory 1003 and send the instructions to the compressor 1001.
[0194] In some embodiments, the interface circuit 1002 executes at least one of the communication steps such as sending and / or receiving in the above method, and the compressor 1001 executes other steps.
[0195] In some embodiments, terms such as interface circuit, interface, transceiver pin, transceiver, etc. may be used interchangeably.
[0196] In some embodiments, chip 1000 further includes one or more memories 1003 for storing instructions. Optionally, all or part of memories 1003 may be outside chip 1000.
[0197] To implement the above embodiments, the present disclosure also proposes a computer-readable storage medium storing a computer program, which when executed by a processor, implements the data compression method as proposed in the foregoing embodiments of the present disclosure.
[0198] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.
[0199] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0200] Any process or method description in a flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present disclosure pertain.
[0201] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing when necessary, and then stored in a computer memory.
[0202] It should be understood that various parts of the present disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0203] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.
[0204] In addition, in various embodiments of the present disclosure, each functional unit may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0205] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.
Claims
1. A data compression method, characterized in that, The method includes: Obtaining a plurality of first strings that meet a preset quantity from the data to be compressed; Based on the compressed characters in the compression cache, respectively matching the plurality of first strings to obtain a first matching result for each of the first strings; Determining target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed.
2. The method according to claim 1, wherein The first matching result includes: The target offset position in the compression cache of the compressed character that successfully matches the first string; The first length corresponding to the continuously identical characters between the characters starting from the first string in the data to be compressed and the characters starting from the target offset position in the compression cache.
3. The method according to claim 2, wherein The step of respectively matching the plurality of first strings based on the compressed characters in the compression cache to obtain a first matching result for each of the first strings includes: Determining a target hash value corresponding to the first string; Based on the target hash value, querying a hash table to obtain the target offset position of the target hash value in the compression cache, where the hash table contains the offset positions in the compression cache of the second strings corresponding to the hash values; Matching the characters starting from the first string in the data to be compressed with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters.
4. The method according to claim 3, wherein After determining the target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed, it further includes: Determining a third string in the data to be compressed corresponding to the target compressed data; Updating the characters in the compression cache based on the third string in a first-in-first-out manner; Updating the hash table based on the offset position of the first string in the compression cache and the corresponding target hash value.
5. The method according to claim 2, wherein The step of determining the target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed includes: Determining a target matching result according to the plurality of first matching results; Replacing the first length of characters starting from the first character string corresponding to the target matching result in the data to be compressed with the target matching result to obtain the target compressed data.
6. The method according to claim 5, characterized in that, The step of determining the target matching result according to the plurality of first matching results includes: Determining the first matching result with the largest first length and greater than the length threshold as the target matching result.
7. The method according to claim 5, characterized in that The step of determining the target matching result according to the plurality of first matching results includes: Determining a second matching result with a first length greater than the length threshold from the plurality of first matching results; In the case where the number of the second matching results is multiple, determining the fourth string corresponding to each of the second matching results in the compression cache; Removing duplicates from the fourth strings in the case where there is partial overlap between two of the fourth strings; Based on the length of the fourth string after deduplication, update the first length in the corresponding second matching result to obtain the target matching result.
8. The method according to claim 1, wherein The obtaining of a plurality of first strings meeting a preset quantity from the data to be compressed includes: Obtain a first quantity of characters starting from the first character of the data to be compressed to obtain a fifth string; Process the fifth string to obtain a plurality of the first strings meeting the preset quantity.
9. A compressor, characterized in that, Includes: A serial-parallel conversion module, a preset quantity of calculation modules, and a first encoder; wherein, The serial-parallel conversion module is used to obtain a plurality of first strings meeting a preset quantity from the data to be compressed; The preset quantity of calculation modules are used to perform matching on the plurality of first strings respectively based on the compressed characters in the compression cache to obtain a first matching result for each of the first strings; The first encoder is used to determine target compressed data according to the obtained plurality of first matching results and the positions of each of the first strings in the data to be compressed.
10. The compressor according to claim 9, characterized in that, The first matching result includes: The target offset position in the compression cache of the compressed characters that match the first string successfully; The first length corresponding to the continuously identical characters between the characters starting from the first string in the data to be compressed and the characters starting from the target offset position in the compression cache.
11. The compressor according to claim 10, characterized in that, The calculation module includes a hash calculation unit, a hash query unit, and a matching unit; The hash calculation unit is used to determine the target hash value corresponding to the first string; The hash query unit is used to query a hash table based on the target hash value to obtain the target offset position of the target hash value in the compression cache, wherein the hash table contains the offset positions in the compression cache of the second strings corresponding to the hash values; The matching unit is used to match the characters starting from the first string in the data to be compressed with the characters starting from the target offset position in the compression cache to obtain the first length corresponding to the continuously identical characters.
12. The compressor according to claim 11, wherein, The compressor further includes a compression cache update module and a hash update module; The compression cache update module is used to determine a third string corresponding to the target compressed data in the data to be compressed and update the characters in the compression cache based on the third string in a first-in, first-out manner; The hash update module is used to update the hash table based on the offset position of the first string in the compression cache and the corresponding target hash value.
13. The compressor according to claim 10, wherein The first encoder is used for: Determine a target matching result according to the plurality of first matching results; Replace the first length of characters starting from the first character corresponding to the target matching result in the data to be compressed with the target matching result to obtain the target compressed data.
14. The compressor according to claim 13, characterized in that, The first encoder is used for: Determine the first matching result with the largest first length and greater than the length threshold as the target matching result.
15. The compressor according to claim 13, characterized in that, The first encoder is used for: Determine a second matching result whose first length is greater than a length threshold from the multiple first matching results; When the number of the second matching results is multiple, determine a fourth string corresponding to each of the second matching results in the compression cache; When there is partial overlap between two fourth strings, deduplicate the fourth strings; Based on the deduplicated fourth strings, update the first length in the corresponding second matching results to obtain the target matching results.
16. The compressor according to claim 9, wherein, The serial-parallel conversion module is configured to: Obtain a first number of characters starting from the first character of the data to be compressed to obtain a fifth string; Process the fifth string to obtain multiple first strings that meet a preset quantity.
17. The compressor according to any one of claims 9-16, characterized in that, The compressor further includes a second encoder, and the second encoder is configured to compress the target compressed data.
18. A data compression device, characterized in that, The device includes: An acquisition module, configured to acquire multiple first strings that meet a preset quantity from the data to be compressed; A matching module, configured to respectively match the multiple first strings based on the compressed characters in the compression cache to obtain a first matching result for each of the first strings; A determination module, configured to determine the target compressed data according to the obtained multiple first matching results and the position of each first string in the data to be compressed.
19. An electronic device, characterized in that, Comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the program, it implements the data compression method according to any one of claims 1-8.
20. A chip, characterized in that, The chip includes a compressor, and the compressor is configured to execute the data compression method according to any one of claims 1-8.
21. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data compression method according to any one of claims 1-8.