Data compression method and device, data decompression method and device and electronic equipment

Abstract

The embodiment of the invention discloses a data compression method and device, a data decompression method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring to-be-compressed data; obtaining a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, the first hash table being a storage structure corresponding to the preset dictionary compression algorithm and including a first byte length, and the second hash table being a storage structure corresponding to the preset dictionary compression algorithm and including a second byte length, the first byte length is greater than the second byte length; performing first matching on the to-be-compressed data according to a first preset byte length; if the first matching is not successful, performing second matching on the to-be-compressed data according to a second preset byte length; and if the second matching is successful, performing compression processing on the to-be-compressed data based on the second hash table to obtain target compressed data. In this way, the compression rate of the preset compression algorithm is improved.

Description

Technical Field

[0001] The present application belongs to the field of data compression technology, and specifically relates to a data compression method, a data decompression method, a device, an electronic device and a storage medium. Background Art

[0002] In the era of rapid development of network communication technology, data lossless compression technology has gradually demonstrated its unique importance, making it widely used in cloud storage and Internet communications, such as compressed file systems, data backup, web front-end services, etc. Data lossless compression technology can reduce data redundancy and reduce the resources occupied by information in storage without affecting the quality of data reconstruction, objectively improving data transmission efficiency. Lossless data compression technology is mainly divided into two categories, namely dictionary-based lossless data compression technology and statistical-based lossless data compression technology. Among them, the compression rate of dictionary-based data compression method needs to be improved when compressing data. Summary of the invention

[0003] In view of the above problems, the present application proposes a data compression method, a data decompression method, a device, an electronic device and a storage medium to improve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a data compression method, the method comprising: obtaining data to be compressed; obtaining a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, the first hash table being a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, the second hash table being a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, the first byte length being greater than the second byte length; performing a first match on the data to be compressed according to the first preset byte length; if the first match is unsuccessful, performing a second match on the data to be compressed according to the second preset byte length; if the second match is successful, compressing the data to be compressed based on the second hash table to obtain target compressed data.

[0005] In a second aspect, an embodiment of the present application provides a data decompression method, the method comprising: reading data stored in the first designated bits of target compressed data; if the hash table corresponding to the data is a first hash table corresponding to a preset dictionary compression algorithm, decompressing the target compressed data based on the first hash table to obtain decompressed data, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm; if the hash table corresponding to the data is a second hash table corresponding to the preset dictionary compression algorithm, decompressing the target compressed data based on the second hash table to obtain decompressed data, the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0006] In a third aspect, an embodiment of the present application provides a data compression device, the device comprising: a data acquisition unit, used to acquire data to be compressed; a hash table acquisition unit, used to acquire a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, the first hash table being a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, the second hash table being a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, the first byte length being greater than the second byte length; a first matching unit, used to perform a first match on the data to be compressed according to a first preset byte length; a second matching unit, used to perform a second match on the data to be compressed according to a second preset byte length if the first match fails; a compression unit, used to compress the data to be compressed based on the second hash table to obtain target compressed data if the second match succeeds.

[0007] In a fourth aspect, an embodiment of the present application provides a data decompression device, the device comprising: a data acquisition unit, used to read the data stored in the first designated bits of the target compressed data; a first decompression unit, used to decompress the target compressed data based on the first hash table to obtain decompressed data if the hash table corresponding to the data is the first hash table corresponding to the preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm; a second decompression unit, used to decompress the target compressed data based on the second hash table to obtain decompressed data if the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0010] The embodiments of the present application provide a data compression method, device, electronic device and storage medium. Obtain the data to be compressed, obtain the first hash table and the second hash table corresponding to the preset dictionary compression algorithm, the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, the first byte length is greater than the second byte length, and then the data to be compressed is first matched according to the first preset byte length. If the first match is not successful, the data to be compressed is second matched according to the second preset byte length. If the second match is successful, the data to be compressed is compressed based on the second hash table to obtain the target compressed data. In the above manner, a second hash table is introduced for the preset dictionary compression algorithm, and the two hash tables are used together to compress the data to be compressed. When the second match is successful, the data to be compressed can be compressed by the second hash table, thereby improving the compression rate of the preset compression algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A flow chart of a data compression method proposed in an embodiment of the present application is shown;

[0013] Figure 2 A schematic diagram of the structure of a first hash table in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of the structure of a second hash table in an embodiment of the present application is shown;

[0015] Figure 4 A flow chart of a data compression method proposed in another embodiment of the present application is shown;

[0016] Figure 5 A schematic diagram showing the structure of a second hash table in another embodiment of the present application is shown;

[0017] Figure 6 A schematic diagram showing the process described in step S210 to step S280 in another embodiment of the present application is shown;

[0018] Figure 7 A flow chart of a data decompression method proposed in an embodiment of the present application is shown;

[0019] Figure 8 A flow chart of a data decompression method proposed in another embodiment of the present application is shown;

[0020] Fig. 9 A schematic diagram showing the process of step S410 to step S430 in another embodiment of the present application is shown;

[0021] Fig.10 A structural block diagram of a data compression device proposed in an embodiment of the present application is shown;

[0022] Fig.11 A structural block diagram of a data decompression device proposed in an embodiment of the present application is shown;

[0023] Fig.12 A structural block diagram of an electronic device for executing a data compression method or a data decompression method according to an embodiment of the present application is shown;

[0024] Fig.13 A storage unit in an embodiment of the present application is shown for storing or carrying program codes for implementing a data compression method or a data decompression method according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] In the era of rapid development of network communication technology, data lossless compression technology has gradually demonstrated its unique importance, making it widely used in cloud storage and Internet communications, such as compressed file systems, data backup, web front-end services, etc. Data lossless compression technology can reduce data redundancy and reduce the resources occupied by information in storage without affecting the quality of data reconstruction, objectively improving data transmission efficiency. Lossless data compression technology is mainly divided into two categories, namely dictionary-based lossless data compression technology and statistical-based lossless data compression technology.

[0027] The inventors found in the study of dictionary-based lossless data compression technology that the principle of dictionary-based lossless data compression technology is to use a small amount of storage structure to store previously existing information. Taking the LZ4 compression algorithm as an example, when compressing data, the corresponding minimum matching length is 4Bytes, so the corresponding minimum storage unit is 3Bytes, otherwise the compression effect cannot be achieved. 4Bytes is equal to 32bits. If the matching of data blocks below 4Bytes (such as 2Bytes or 3Bytes) is not considered, there will be a lot of repeated data below 4Bytes (such as 2Bytes or 3Bytes) that cannot be compressed, so a good compression rate cannot be achieved.

[0028] Therefore, the inventors proposed the data compression method, data decompression method, device, electronic device and storage medium in the present application. Obtain the data to be compressed, obtain the first hash table and the second hash table corresponding to the preset dictionary compression algorithm, the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, the first byte length is greater than the second byte length, and then the data to be compressed is first matched according to the first preset byte length. If the first match is not successful, the data to be compressed is second matched according to the second preset byte length. If the second match is successful, the data to be compressed is compressed based on the second hash table to obtain the target compressed data. In the above manner, a second hash table is introduced for the preset dictionary compression algorithm, and the two hash tables are used together to compress the data to be compressed. When the second match is successful, the data to be compressed can be compressed by the second hash table, thereby improving the compression rate of the preset compression algorithm.

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0030] See also Figure 1 , a data compression method provided in an embodiment of the present application is applied to an electronic device, the method comprising:

[0031] Step S110: Obtain data to be compressed.

[0032] In the embodiment of the present application, the data to be compressed may be data that is determined to be compressed. The data to be compressed may be any type of data that needs to be compressed, for example, the data to be compressed may be data that needs to be written into a memory, the data to be compressed may be data generated by a processor when executing a program, or may be data received by interacting with other devices or processes, or may be image data or text data sent by other electronic devices, or may be data exchanged between multiple memories, etc., which are not specifically limited here.

[0033] As a method, the data to be compressed can be the data written into the memory for the first time. Memory (also known as random access memory, RAM) refers to the memory used when running a program (i.e., running memory). It can only temporarily store data and is used to exchange cache data with the processor, but the memory itself cannot be used to store data for a long time. When the application is running, it needs to occupy memory space to store data during the running process. The size of the memory space occupied by different applications in different running states may not be the same.

[0034] The data written into the memory for the first time may be hot data in the time series data, which refers to time series data. Time series data is a data column recorded in chronological order according to a unified indicator, and the time series data may include time information, data type, and data information. The time information may specifically be a timestamp; the data type may be a known data type or a data type that may appear in the future, such as existing data types that may include Integer, Long, Float, Double, String, BinaryArray, Image, and other types; the data information may be the data itself, that is, the original data that does not contain descriptions such as time information and data type.

[0035] Since time series data has a massive nature, it is impossible to store all the data in the memory of an electronic device. In order to solve this problem, in the embodiment of the present application, the data in the current time is placed in the memory as hot data, and then when the hot data exceeds the specified time, the electronic device will compress it and finally persist it as cold data. Among them, hot data can be frequently accessed data, which is usually stored in a cache storage device, and cold data refers to data with low access frequency and low access performance requirements, such as data that will not be accessed in the short term or even no longer needed in the future.

[0036] As another method, the data to be compressed may be a determined allocated memory. When performing memory compression on the allocated memory, a portion of the allocated memory may be selected from all the allocated memory, and then the portion of the allocated memory may be firstly subjected to memory compression by a hardware-based compression algorithm. The size of the portion of the allocated memory may be determined according to a preset compression amount. The preset compression amount is a preset size of the allocated memory that needs to be subjected to memory compression.

[0037] Optionally, in an embodiment of the present application, when the kernel allocates memory to a process, it is known how much memory each process needs, and then when allocating memory to the corresponding process, it can record how much memory is allocated to each process. Therefore, when the allocated memory is compressed according to the preset compression amount, part of the allocated memory can be selected from all the allocated memory for memory compression according to the size of the memory corresponding to each process and the preset compression amount. Exemplarily, if the preset compression amount is 10MB, and the size of the memory corresponding to process 1 is 2MB, the size of the memory corresponding to process 2 is 5MB, the size of the memory corresponding to process 3 is 8MB, and the size of the memory corresponding to process 4 is 3MB, then when part of the allocated memory is selected from all the allocated memory for memory compression according to the size of the memory corresponding to each process and the preset compression amount, memory compression can be selected for the memory corresponding to process 1, process 2 and process 4; or memory compression can be selected for the memory corresponding to process 1 and process 3.

[0038] Optionally, in the embodiment of the present application, the data to be compressed in the memory may be obtained once every set time period. Alternatively, the operation of obtaining the data to be compressed currently stored in the memory may be performed according to a trigger instruction input by the user.

[0039] Step S120: Obtain a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0040] In an embodiment of the present application, the preset dictionary compression algorithm is a compression algorithm pre-set for compressing data to be compressed. The specific working principle of the preset dictionary compression algorithm is to encode a long string (also referred to as a match or phrase) into a short mark, and replace the phrase in the dictionary with a small mark, that is, it achieves the purpose of data compression by replacing a long string that appears repeatedly in the data with a small mark. The symbols it processes are not necessarily text characters, but can also be other symbols of any size. Among them, the preset dictionary compression algorithm can be an LZ4 compression algorithm, an LZ77 compression algorithm, an LZ78 compression algorithm, an LZSS compression algorithm, and an LZW compression algorithm, etc., which are not specifically limited here.

[0041] The first hash table and the second hash table may be hash tables pre-set for a preset dictionary compression algorithm, the first hash table is a storage structure after modifying the original storage structure of the preset dictionary compression algorithm, and the second hash table is a new storage structure introduced by the preset dictionary compression algorithm. The target compressed data obtained after compression by the second hash table is smaller than the target compressed data obtained after data compression by the first hash table. The first hash table is a storage structure obtained by using the first designated bit of the original storage structure of the preset dictionary compression algorithm as a flag bit, and keeping the remaining structure unchanged.

[0042] Exemplarily, if the first hash table is a storage structure including a length of 3Bytes, then the second hash table can be a storage structure including a length of 2Bytes or 1Byte. The first hash table is used to store data after matching and compression according to the first preset byte length, and the second hash table is used to store data after matching and compression according to the second preset byte length. The first preset byte length can be understood as the minimum matching length corresponding to the first hash table, that is, the minimum matching length originally corresponding to the preset dictionary compression algorithm, and the second preset byte length can be understood as the minimum matching length corresponding to the second hash table, that is, the minimum matching length corresponding to the introduced second storage structure. Exemplarily, if the preset dictionary compression algorithm is the LZ4 compression algorithm, then the first preset byte length can be 4Bytes, and the second preset byte length can be 3Bytes or 2Bytes. In the case where the preset dictionary compression algorithm is the LZ4 compression algorithm, the first hash table can be as follows Figure 2 As shown, Figure 2 The storage structure shown is based on the original storage structure of the LZ4 compression algorithm. The main idea is to take the first bit of the original storage structure of the LZ4 compression algorithm as the flag bit F, whose value is 0 or 1, to distinguish whether the current storage structure is used Figure 2 The storage structure is still Figure 3 The storage structure of the second hash table can be as follows Figure 3 As shown, Figure 3 The storage structure in occupies 2 bytes. Its first bit is the flag bit, which has the same function as Figure 2 The flag bit in has the same function, and its remaining 15 bits are used to store the offset D of the matched 3-Bytes character (indicating the distance from the previous matching character). The 15 bits here can represent a maximum matching length of 32767, which means that it can be used not only in ordinary 4096 memory page compression, but also in compression based on 16k and 32k large memory pages, or in compression of general data.

[0043] As a method, the first hash table and the second hash table can be pre-set and stored in a preset storage area. When the data to be compressed is obtained, the first hash table and the second hash table corresponding to the preset dictionary compression algorithm can be directly obtained from the preset storage area.

[0044] Step S130: performing a first match on the data to be compressed according to a first preset byte length.

[0045] In the embodiment of the present application, the first preset byte length is the minimum matching length corresponding to the first hash table, that is, the minimum matching length originally corresponding to the preset dictionary compression algorithm.

[0046] After obtaining the data to be compressed, the data to be compressed is first matched according to the original minimum matching length of the preset dictionary compression algorithm. For example, if the preset dictionary compression algorithm is the LZ4 compression algorithm, then the minimum matching length corresponding to the preset dictionary compression algorithm is 4Bytes, and the data to be compressed is matched according to 4Bytes, that is, in the process of compressing the data to be compressed, at least 4Bytes are used as the scanning window to search for matches, and 1Byte is moved each time for scanning, and compression is performed when duplicates are encountered.

[0047] Step S140: If the first matching is unsuccessful, a second matching is performed on the data to be compressed according to a second preset byte length.

[0048] In the embodiment of the present application, the second preset byte length can be understood as the minimum matching length corresponding to the second hash table, that is, the minimum matching length corresponding to the second storage structure introduced. Among them, the first preset byte length is greater than the second preset byte length. Exemplarily, if the byte length included in the introduced second storage structure is 2Bytes, then the minimum matching length corresponding to the storage structure is 3Bytes.

[0049] As a way, the failure of the first match indicates that the data length of the data to be compressed is lower than the first preset byte length. When it is determined that the first match is unsuccessful, it can be switched to match according to the second preset byte length, that is, it can be switched to a smaller match length to match the data to be compressed. Exemplarily, if the first match is to match the data to be compressed according to a length of 4Bytes, and the second match is to match the data to be compressed according to a length of 3Bytes, when the matching of the compressed data to be processed according to the length of 4Bytes is unsuccessful, it can be attempted to match the data to be compressed according to the length of 3Bytes.

[0050] Step S150: If the second match is successful, compress the data to be compressed based on the second hash table to obtain target compressed data.

[0051] In the embodiment of the present application, if the second match is successful, the data to be compressed is stored according to the second hash table, and the offset of the matching characters in the data to be compressed is stored in the second hash table.

[0052] The present application provides a data compression method, which obtains data to be compressed, obtains a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, and then performs a first match on the data to be compressed according to a first byte length. If the first match fails, the data to be compressed is second matched according to the second byte length. If the second match succeeds, the data to be compressed is compressed based on the second hash table to obtain target compressed data. In the above manner, a second hash table is introduced for the preset dictionary compression algorithm, and the two hash tables are used together to compress the data to be compressed. When the second match succeeds, the data to be compressed can be compressed by the second hash table, thereby improving the compression rate of the preset compression algorithm.

[0053] See also Figure 4 , a data compression method provided in an embodiment of the present application is applied to an electronic device, the method comprising:

[0054] Step S210: Obtain data to be compressed.

[0055] Step S220: Obtain a first hash table and a second hash table corresponding to a preset dictionary compression algorithm.

[0056] Among them, the data stored in the first designated bits of the first hash table are used as the first flag bit to identify that the data to be compressed is compressed based on the first hash table; the data stored in the first designated bits of the second hash table are used as the second flag bit to identify that the data to be compressed is compressed based on the second hash table.

[0057] In the embodiment of the present application, the first designated bit is a bit pre-set to store an identifier for distinguishing different storage structures. The number of storage structures to be distinguished is different, and the number of corresponding first designated bits is different. For example, if the number of storage structures to be distinguished is 2, then the number of first designated bits is 1, that is, two different storage structures can be distinguished by 0 or 1; if the number of storage structures to be distinguished is 3 or 4, then the number of first designated bits can be 2, that is, 3 or 4 storage structures can be distinguished by 00, 01, 10 and 11; if the number of storage structures to be distinguished is greater than 4, then the number of first designated bits will be greater than 2, and will not be repeated here.

[0058] As a way, the number of the second hash table can be more than one. When the minimum matching length corresponding to the original storage structure of the preset dictionary compression algorithm is greater than 3Bytes, the number of the second hash table can include at least two. For example, if the preset dictionary compression algorithm is the LZ4 compression algorithm, and the minimum matching length corresponding to the original storage structure of the LZ4 compression algorithm is 4Bytes, then the second hash table can include two, one of which is as follows: Figure 3 The storage structure shown is a storage structure including 2Bytes length. One is as follows Figure 5 The storage structure shown is a storage structure including a length of 1 Byte. The storage structure is adapted to match 2-Byte characters. Since a 2-Byte character string is to be compressed, D in the second hash table is changed to 7 bits, that is, the maximum matching length is 127. In the corresponding compression process, when a 2-Byte compressed block is matched, a judgment is also made. If the distance to the matching character in the dictionary is greater than 127, compression is abandoned.

[0059] In the embodiment of the present application, when the number of second hash tables may be more than one, a hash table may be selected from multiple second hash tables as the hash table in the embodiment of the present application. When selecting a hash table from multiple second hash tables, it may be determined according to the performance loss corresponding to each hash table, and the hash table with the smallest corresponding performance loss may be selected as the second hash table in the embodiment of the present application. The performance loss refers to the performance loss of the electronic device when compressing the data to be compressed.

[0060] Optionally, the second hash table may be composed of two fields, one field being a flag bit and the other field being a matching character offset, wherein the flag bit is used to store data representing different storages; and the matching character offset is used to store the matching character offset in the data to be compressed.

[0061] Step S230: Perform a first match on the data to be compressed according to a first preset byte length to determine whether the first match is successful; if the match is successful, execute step S240; if the match is not successful, execute step S250.

[0062] In the embodiment of the present application, when the first match is successful, the match is expanded according to the original compression process of the preset dictionary compression algorithm, and the data to be compressed is stored according to the first hash table.

[0063] Step S240: compressing the data to be compressed based on the first hash table to obtain target compressed data.

[0064] Step S250: Perform a second match on the data to be compressed according to the second preset byte length to determine whether the second match is successful; if the match is successful, execute step S260; if the match is not successful, execute steps S270 and S280.

[0065] Step S260: compressing the data to be compressed based on the second hash table to obtain target compressed data.

[0066] Step S270: Update the first hash table and the second hash table.

[0067] In the embodiment of the present application, when the second match fails, the first hash table and the second hash table are updated. That is, if the second match fails, according to the original compression process of the preset dictionary compression algorithm, the hash information of the current first preset byte length is stored in the first hash table, and the hash information of the current second preset byte length is stored in the second hash table, and then the current position pointer continues to slide forward 1 Byte, and the above process is repeated. Among them, the current position pointer refers to the position of the starting character of the data to be compressed that is currently matched.

[0068] Step S280: performing a first match on the data to be compressed according to the first preset byte length corresponding to the updated first hash table.

[0069] In the embodiment of the present application, after the first hash table and the second hash table are updated and the current position pointer is slid forward by 1 Byte, the data to be compressed is re-matched according to the first preset byte length.

[0070] Exemplarily, the process described in step S210 to step S280 can be as follows: Figure 6 As shown, the preset dictionary compression algorithm is the LZ4 compression algorithm, and the first hash table is as follows Figure 2 As shown, the second hash table is as follows Figure 3As shown. At the beginning of compression, two hash tables are initialized: LZ4_hash and LZ3_hash; then the data to be compressed is matched according to 4Bytes to determine whether it matches. If it matches, the match is expanded according to the original LZ4 compression algorithm process, and the corresponding offset is stored according to the storage structure b1 (equivalent to the first hash table in this application); if it does not match, try 3Bytes matching. If the match is successful, the corresponding offset is stored according to b2 (equivalent to the second hash table in this application). If the match fails, the current 4Bytes hash information is stored in the table lz4_hash and the current 3Bytes hash information is stored in the table lz3_hash according to the original lz4 method, and then the current position pointer continues to slide 1Byte, and the above process is repeated.

[0071] The present application provides a data compression method, firstly obtaining the data to be compressed, obtaining the first hash table and the second hash table corresponding to the preset dictionary compression algorithm, and then performing a first match on the data to be compressed according to the first byte length, determining whether the first match is successful, and if the match is successful, compressing the data to be compressed based on the first hash table to obtain the target compressed data; if the match is not successful, matching the data to be compressed according to the second byte length, determining whether the second match is successful, and if the match is successful, compressing the data to be compressed based on the second hash table, and if the match is not successful, sliding the pointers of the first hash table and the second hash table forward by 1 byte length to update the first hash table and the second hash table, and finally performing a first match on the data to be compressed according to the first byte length included in the updated first hash table. Through the above method, a second hash table is introduced for the preset dictionary compression algorithm, and the two hash tables are used together to compress the data to be compressed. When the second match is successful, the data to be compressed can be compressed through the second hash table, thereby improving the compression rate of the preset compression algorithm.

[0072] See also Figure 7 , a data decompression method provided in an embodiment of the present application is applied to an electronic device, the method comprising:

[0073] Step S310: Read the data stored in the first designated bits of the target compressed data.

[0074] In the embodiment of the present application, the data stored in the first designated bit position is data representing different storage structures.

[0075] As a method, the data stored in the first designated bits of the target compressed data is read to determine the storage structure corresponding to the target compressed data. The number of the first designated bits can be determined according to the corresponding preset dictionary compression algorithm. The number of the first designated bits corresponding to different preset dictionary compression algorithms can be different. When the data to be compressed is compressed by the preset dictionary compression algorithm, when the compressed data is stored, the identifier of the preset dictionary compression algorithm corresponding to the data to be compressed can be stored at the same time, so that when the target compressed data is decompressed, the number of the first designated bits can be determined according to the identifier of the preset dictionary compression algorithm.

[0076] Step S320: If the hash table corresponding to the data is the first hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the first hash table to obtain decompressed data, wherein the first hash table is a storage structure corresponding to the preset dictionary compression algorithm and includes a first byte length.

[0077] In an embodiment of the present application, if the hash table corresponding to the data stored in the previous designated bit is the first hash table corresponding to the preset dictionary compression algorithm, it means that the current target compressed data is stored according to the first hash table, and the target compressed data can be decompressed according to the decompression method corresponding to the preset dictionary compression algorithm.

[0078] Step S330: If the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the second hash table to obtain decompressed data, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0079] In an embodiment of the present application, if the hash table corresponding to the data stored in the previously specified bit position is the second hash table corresponding to the preset dictionary compression algorithm, it means that the current target compressed data is stored according to the second hash table, and the data stored in other byte positions of the second hash table can continue to be read to obtain the offset information. After obtaining the offset information, the data of the second preset byte length can be copied from the corresponding place to the current position to complete the decompression of the target compressed data.

[0080] The present application provides a data decompression method, which reads the data stored in the first designated bits of the target compressed data, and if the hash table corresponding to the data is the first hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the first hash table to obtain the decompressed data; if the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the second hash table to obtain the decompressed data. Through the above method, when the data stored in the first designated bits of the target compressed data are different, the target compressed data can be decompressed based on different hash tables, thereby improving the flexibility of decompressing the target compressed data.

[0081] See also Figure 8 , a data decompression method provided in an embodiment of the present application is applied to an electronic device, the method comprising:

[0082] Step S410: Read the data in the first byte of the target compressed data.

[0083] In the embodiment of the present application, when decompressing the target compressed data, the data of the first byte of the target compressed data is read first, that is, the data stored in the first Byte is read first.

[0084] Step S420: Read the data of the first designated bit from the data of the first byte.

[0085] In the embodiment of the present application, after reading the data stored in the first Byte, the first designated bit is separated from the data stored in the first Byte to obtain a flag bit corresponding to the target compressed data.

[0086] Step S430: If the hash table corresponding to the data is the first hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the first hash table to obtain decompressed data, wherein the first hash table is a storage structure corresponding to the preset dictionary compression algorithm and includes a first byte length.

[0087] Step S440: If the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the second hash table to obtain decompressed data, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0088] In an embodiment of the present application, if the hash table corresponding to the data stored in the previous designated bit is the second hash table corresponding to the preset dictionary compression algorithm, it means that the current target compressed data is stored according to the second hash table, then continue to read the data stored in the remaining bytes, and combine it with the data in the first Byte to obtain the offset information corresponding to the target compressed data. After obtaining the offset information corresponding to the target compressed data, the data of the second preset byte length can be copied from the corresponding place to the current position to complete the decompression of the target compressed data.

[0089] Exemplarily, the process described in step S410 to step S430 can be as follows: Fig. 9 As shown, the first hash table is as follows Figure 2 As shown, the second hash table is as follows Figure 3 As shown. Decompression starts, read the first Byte, and obtain the first bit F. If F is 1, it means that the current compressed block is in the b2 mode (such as Figure 2 If the current compressed block is stored in the manner shown in the figure, then continue to read 1 Byte (a total of 2 Bytes are read), and after obtaining the offset information, copy 3 Bytes from the corresponding place to the current position to complete the decompression of the current block (equivalent to the target compressed data in this application); if F is not 1, it means that the current compressed block is stored in the manner b1, and then it is decompressed according to the decompression steps of the LZ4 compression algorithm to complete the decompression of the current block (equivalent to the target compressed data in this application).

[0090] The present application provides a data decompression method, which can decompress the target compressed data based on different hash tables when the data stored in the first designated bits of the target compressed data are different, thereby improving the flexibility of decompressing the target compressed data.

[0091] See also Fig.10 , an embodiment of the present application provides a data compression device 500, the device 500 comprising:

[0092] The data acquisition unit 510 is used to acquire data to be compressed.

[0093] The hash table acquisition unit 520 is used to obtain a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0094] Among them, the data stored in the first designated bits of the first hash table are used as the first flag bit to identify that the data to be compressed is compressed based on the first hash table; the data stored in the first designated bits of the second hash table are used as the second flag bit to identify that the data to be compressed is compressed based on the second hash table.

[0095] The first matching unit 530 is configured to perform a first match on the data to be compressed according to a first preset byte length.

[0096] As a method, the first matching unit 530 is specifically used to update the first hash table and the second hash table if the second matching is unsuccessful; and perform a first match on the data to be compressed according to a first preset byte length corresponding to the updated first hash table.

[0097] The second matching unit 540 is configured to perform a second matching on the data to be compressed according to a second preset byte length if the first matching fails.

[0098] The compression unit 550 is configured to compress the data to be compressed based on the second hash table to obtain target compressed data if the second match is successful.

[0099] As a method, the compression unit 550 is specifically configured to compress the data to be compressed based on the first hash table to obtain target compressed data if the first match is successful.

[0100] See also Fig.11 , an embodiment of the present application provides a data decompression device 600, the device 600 comprising:

[0101] The data acquisition unit 610 is used to read the data stored in the first designated bits of the target compressed data.

[0102] As a method, the data acquisition unit 610 is specifically used to read the data in the first byte of the target compressed data; and read the data of the first designated bit from the data of the first byte.

[0103] The first decompression unit 620 is used to decompress the target compressed data based on the first hash table to obtain decompressed data if the hash table corresponding to the data is the first hash table corresponding to the preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm.

[0104] The second decompression unit 630 is used to decompress the target compressed data based on the second hash table to obtain decompressed data if the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

[0105] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0106] The following will be combined Fig.12 An electronic device provided by the present application is described.

[0107] See also Fig.12 Based on the above data compression method or data decompression method and device, the embodiment of the present application also provides another electronic device 800 that can execute the above data compression method or data decompression method. The electronic device 800 includes one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 that are coupled to each other. Among them, the memory 804 stores a program that can execute the content of the above embodiments, and the processor 802 can execute the program stored in the memory 804.

[0108] Among them, the processor 802 may include one or more processing cores. The processor 802 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 804, and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 802 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 802, but may be implemented separately through a communication chip.

[0109] The memory 804 may include a random access memory (RAM) or a read-only memory (ROM). The memory 804 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 800 during use.

[0110] The network module 806 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with electronic devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 806 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 may exchange information with a base station.

[0111] Please refer to Fig.13 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0112] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0113] The present application provides a data compression method or data decompression method, device, electronic device and storage medium, which obtains the data to be compressed, obtains the first hash table and the second hash table corresponding to the preset dictionary compression algorithm, the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, the first byte length is greater than the second byte length, and then the data to be compressed is first matched according to the first preset byte length, if the first match is not successful, the data to be compressed is second matched according to the second preset byte length, if the second match is successful, the data to be compressed is compressed based on the second hash table to obtain the target compressed data. In the above manner, a second hash table is introduced for the preset dictionary compression algorithm, and the two hash tables are used together to compress the data to be compressed. When the second match is successful, the data to be compressed can be compressed by the second hash table, thereby improving the compression rate of the preset compression algorithm.

[0114] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A data compression method, characterized in that: The method comprises: Get the data to be compressed; Obtain a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length; Performing a first match on the data to be compressed according to a first preset byte length; If the first matching fails, performing a second matching on the data to be compressed according to a second preset byte length; If the second match is successful, the data to be compressed is compressed based on the second hash table to obtain target compressed data.

2. The method according to claim 1, characterized in that After performing a first match on the data to be compressed according to the first preset byte length, the method further includes: If the first match is successful, the data to be compressed is compressed based on the first hash table to obtain target compressed data.

3. The method according to claim 1, characterized in that If the first matching fails, performing a second matching on the data to be compressed according to a second preset byte length also includes: If the second match fails, updating the first hash table and the second hash table; The first matching of the to-be-compressed data according to a first preset byte length includes: The data to be compressed is first matched according to the first preset byte length corresponding to the updated first hash table.

4. The method according to claim 1 or 3, characterized in that: The data stored in the first designated bit of the first hash table is used as a first flag bit to identify that compression processing is performed on the data to be compressed based on the first hash table; The data stored in the first designated bit of the second hash table is used as a second flag bit to identify that compression processing is performed on the data to be compressed based on the second hash table.

5. A data decompression method, characterized in that: The method comprises: Read the data stored in the first specified bit position of the target compressed data; If the hash table corresponding to the data is a first hash table corresponding to a preset dictionary compression algorithm, decompressing the target compressed data based on the first hash table to obtain decompressed data, wherein the first hash table is a storage structure corresponding to the preset dictionary compression algorithm and includes a first byte length; If the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, the target compressed data is decompressed based on the second hash table to obtain decompressed data, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

6. The method according to claim 5, characterized in that The step of reading the data stored in the first designated bit of the target compressed data includes: Reading data in the first byte of the target compressed data; From the first byte of data, read the data of the first specified bit.

7. A data compression device, characterized in that: The device comprises: A data acquisition unit, used for acquiring data to be compressed; a hash table acquisition unit, configured to acquire a first hash table and a second hash table corresponding to a preset dictionary compression algorithm, wherein the first hash table is a storage structure including a first byte length corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length; A first matching unit, configured to perform a first match on the data to be compressed according to a first preset byte length; A second matching unit, configured to perform a second matching on the data to be compressed according to a second preset byte length if the first matching fails; A compression unit is used to compress the data to be compressed based on the second hash table to obtain target compressed data if the second match is successful.

8. A data decompression device, characterized in that: The device comprises: A data acquisition unit, used for reading data stored in the first designated bit position of the target compressed data; A first decompression unit is configured to, if the hash table corresponding to the data is a first hash table corresponding to a preset dictionary compression algorithm, decompress the target compressed data based on the first hash table to obtain decompressed data, wherein the first hash table is a storage structure corresponding to the preset dictionary compression algorithm and includes a first byte length; A second decompression unit is used to decompress the target compressed data based on the second hash table to obtain decompressed data if the hash table corresponding to the data is the second hash table corresponding to the preset dictionary compression algorithm, and the second hash table is a storage structure including a second byte length corresponding to the preset dictionary compression algorithm, and the first byte length is greater than the second byte length.

9. An electronic device, characterized in that: The method comprises one or more processors; one or more programs are stored in the memory and configured to execute the method according to any one of claims 1 to 7 by the one or more processors.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images