Data compression method and device, data decompression method and device, equipment and storage medium

Abstract

The invention discloses a data compression method which is used for solving the problem that an existing data compression scheme is low in compression efficiency. The method comprises the following steps: performing block processing on original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; according to the coding dictionary, searching for a matching character string in each data block, and determining a matching position and a character string length corresponding to the matching character string; according to a coding dictionary, the matching position and the character string length, respectively coding each data block to obtain a first coding character string corresponding to the data block; encoding the matching position and the character string length to obtain a second encoded character string corresponding to the data block; according to the first coding character string and the second coding character string, respectively performing prediction compression coding on the data blocks to obtain block compression data corresponding to the data blocks; and combining the block compressed data to obtain compressed data corresponding to the original data.

Description

Technical Field

[0001] The present application relates to the field of data compression technology, and in particular to a data compression method, a data decompression method, an apparatus, a device, a storage medium, and a computer program product. Background Art

[0002] With the rapid development of Internet technology and cloud computing technology, services and applications around networks and data have experienced explosive growth. While facilitating users' daily production and life, users also generate massive amounts of data when using these services and applications. The storage and transmission costs of these massive amounts of data are very high. In order to reduce the loss of data transmission and storage resources, the current common solution is to compress the massive data before transmitting and storing it.

[0003] Data compression refers to a technical method that reduces the amount of data to reduce storage space, improves its transmission, storage and processing efficiency, or reorganizes data according to a certain algorithm to reduce data redundancy and storage space without losing useful information.

[0004] Existing data compression schemes are mainly implemented by encoding and decoding the original data. Specifically, during the data compression process, the sender uses a compression algorithm to encode the original data and convert it into a smaller representation. After receiving the compressed data, the receiver uses the same compression algorithm to decode it to restore the original data. Common data compression algorithms include Huffman coding algorithm, LZ77 coding algorithm, etc. These algorithms use different strategies to compress data according to the characteristics and structure of the data to achieve the purpose of reducing storage and transmission costs.

[0005] However, in the process of encoding and compression, the existing data compression algorithm needs to calculate the characters or symbols corresponding to the original data in sequence, resulting in a slow compression speed and a low data compression rate.

[0006] It can be seen that how to improve data compression efficiency while improving data compression rate has become a problem that needs to be solved urgently. Summary of the invention

[0007] The embodiment of the present application provides a data compression method to solve the problem of low compression efficiency in existing data compression solutions.

[0008] The embodiment of the present application also provides a data decompression method to solve the problem of low compression efficiency in existing data compression schemes.

[0009] The embodiment of the present application also provides a data compression device to solve the problem of low compression efficiency in existing data compression solutions.

[0010] The embodiment of the present application also provides a data compression device to solve the problem of low compression efficiency in existing data compression solutions.

[0011] The embodiment of the present application also provides a computer-readable storage medium to solve the problem of low compression efficiency in existing data compression solutions.

[0012] The embodiment of the present application also provides a computer program product to solve the problem of low compression efficiency in existing data compression solutions.

[0013] The present application embodiment adopts the following technical solutions:

[0014] A data compression method comprises: performing block processing on original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; searching for matching strings in each of the data blocks according to the coding dictionary, and determining the matching position and string length corresponding to the matching string; encoding each of the data blocks according to the coding dictionary, the matching position and the string length to obtain a first coding string corresponding to the data block; encoding the matching position and the string length to obtain a second coding string corresponding to each of the data blocks; performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to each of the data blocks; and combining the block compressed data to obtain compressed data corresponding to the original data.

[0015] A data decompression method comprises: decompressing compressed data to be decompressed, obtaining encoding dictionary information, encoding character string and encoding mode information corresponding to the compressed data; constructing an empty decoding dictionary database according to the encoding dictionary information; decoding the encoding character string according to the encoding mode information to obtain a matching position, a character string length and an undecoded character string; decoding the undecoded character string according to the encoding dictionary information, the matching position and the character string length to obtain original data corresponding to the compressed data.

[0016] A data compression device comprises: a block division unit, used for performing block processing on original data to obtain at least two data blocks; a dictionary construction unit, used for constructing a coding dictionary corresponding to the original data; an encoding unit, used for encoding each of the data blocks according to the coding dictionary, to obtain a first coding string corresponding to the data block; a matching unit, used for searching for a matching string in each of the data blocks according to the coding dictionary, and determining a matching position and a string length corresponding to the matching string; an encoding unit, used for encoding the matching position and the string length, to obtain a second coding string corresponding to each of the data blocks; a compression unit, used for performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string, to obtain block compressed data corresponding to each of the data blocks; and a compression unit, used for combining the block compressed data, to obtain compressed data corresponding to the original data.

[0017] A data decompression device comprises: a decompression unit, used to decompress compressed data to be decompressed, and obtain encoding dictionary information, encoding character string and encoding mode information corresponding to the compressed data; a dictionary construction unit, used to construct an empty decoding dictionary database according to the encoding dictionary information; a decoding unit, used to decode the encoding character string according to the encoding mode information, and obtain a first decoded character string and an undecoded character string; a decoding unit, used to decode the undecoded character string according to a range encoding method, and obtain a matching position and a character string length; a decoding unit, used to perform secondary decoding on the first decoded character string according to the encoding dictionary information, the matching position and the character string length, and obtain original data corresponding to the compressed data.

[0018] A data compression device comprises: a processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor performs the following operations: block processing of original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; searching for matching strings in each of the data blocks according to the coding dictionary, and determining the matching position and string length corresponding to the matching string; encoding each of the data blocks according to the coding dictionary, the matching position and the string length to obtain a first coding string corresponding to the data block; encoding the matching position and the string length to obtain a second coding string corresponding to each of the data blocks; performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to each of the data blocks; and combining the block compressed data to obtain compressed data corresponding to the original data.

[0019] A computer-readable storage medium stores one or more programs, which, when executed by an electronic device including multiple application programs, enable the electronic device to perform the following operations: block processing of original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; searching for matching strings in each of the data blocks according to the coding dictionary, and determining matching positions and string lengths corresponding to the matching strings; encoding each of the data blocks according to the coding dictionary, the matching positions and the string lengths to obtain first coding strings corresponding to the data blocks; encoding the matching positions and the string lengths to obtain second coding strings corresponding to the data blocks; performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to each of the data blocks; and combining the block compressed data to obtain compressed data corresponding to the original data.

[0020] A computer program product comprises a computer program, which, when executed by a processor, implements the following steps: performing block processing on original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; searching for matching strings in each of the data blocks according to the coding dictionary, and determining matching positions and string lengths corresponding to the matching strings; encoding each of the data blocks according to the coding dictionary, the matching positions and the string lengths to obtain first coding strings corresponding to the data blocks; encoding the matching positions and the string lengths to obtain second coding strings corresponding to the data blocks; performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to the data blocks; and combining the block compressed data to obtain compressed data corresponding to the original data.

[0021] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0022] According to a data compression method provided by an embodiment of the present application, before data transmission, the communication system can block the original data to be transmitted to obtain at least two data blocks, and encode each of the data blocks by constructing a coding dictionary to obtain a first coding string corresponding to the data block, and at the same time, according to the coding dictionary, find the matching string in each of the data blocks, and determine the matching position and string length corresponding to the matching string, and use the range coding method to perform secondary coding on the matching position and string length to obtain a second coding string, and finally, use the predictive compression coding method to compress the first coding string and the second string to obtain the final compressed data. According to the data compression method provided by the present scheme, since multiple coding methods (coding dictionary, range coding and predictive compression coding) are used to encode and compress the original data in turn, compared with the existing compression scheme that only uses a single coding method, the method provided by the present scheme can achieve high compression rate data compression, which improves the data compression efficiency on the one hand, and on the other hand, by transmitting compressed data, it greatly reduces the occupation of bandwidth resources in the data transmission process, and improves the data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A schematic diagram of a specific process of a data compression method provided in an embodiment of the present application;

[0025] Figure 2 A specific flow chart of a data decompression method provided in an embodiment of the present application

[0026] Figure 3 A schematic diagram of the specific structure of a data compression device provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the specific structure of a data decompression device provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the specific structure of a data compression device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0030] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0031] A data compression method provided in an embodiment of the present application is used to solve the problem of low compression efficiency in existing data compression schemes.

[0032] The executor of the data compression method provided in the embodiment of the present application may be, but is not limited to, at least one of a communication server, a data transmission server, a data storage server, a video server, and an audio server; in addition, the executor of the method may also be an application (Application, APP) running on these servers or the system itself.

[0033] For ease of description, the following takes the method being performed by a communication system for data transmission as an example to introduce the implementation of the method. It can be understood that the method being performed by a communication system is only an exemplary description and should not be understood as a limitation on the method.

[0034] The specific implementation flow chart of the data compression method provided in this application is as follows Figure 1 As shown, it mainly includes the following steps:

[0035] Step 11, dividing the original data into blocks to obtain at least two data blocks;

[0036] The original data refers to the data to be transmitted obtained by the communication system. In order to improve the data transmission efficiency and reduce the bandwidth resources occupied by the data transmission process, the communication system will compress the original data to be transmitted before data transmission.

[0037] In order to avoid the problem that it takes a long time to directly encode and compress the original data due to the large amount of data, in an embodiment of the present application, the original data can be divided into multiple data blocks, and then when the data is subsequently encoded and compressed, the communication system can encode and compress these data blocks separately at the same time, thereby improving the data compression efficiency.

[0038] Step 12, constructing a coding dictionary corresponding to the original data;

[0039] In the embodiment of the present application, the communication system may specifically construct a coding dictionary according to the following sub-steps, including:

[0040] Sub-step 1201, constructing an empty original encoding dictionary;

[0041] Sub-step 1202, initializing the original encoding dictionary constructed by executing sub-step 1201;

[0042] Specifically, the initialization operation of the original encoding dictionary includes:

[0043] 1. Create a coding dictionary database (or coding dictionary data table), wherein the constructed coding dictionary database (or coding dictionary data table) usually includes at least three fields: a reference tag, a sample string, a hash value or other function fields for retrieving the coding dictionary database.

[0044] 2. Preset an initial string, which can have any initial value, such as "space" and so on.

[0045] Sub-step 1203, scanning the original data, obtaining at least one character from the original data in sequence, and obtaining a character string to be processed.

[0046] In the embodiment of the present application, the communication system reads all characters contained in the original data in sequence by scanning, and adds the read characters to the data buffer area for subsequent processing.

[0047] Sub-step 1204, determining whether the character string to be processed has a matching substring in the encoding dictionary database;

[0048] Specifically, when the communication system scans the original data to be transmitted for the first time, the communication system can read the first character in the original data, and add the read character to the data buffer area, and at the same time search whether there is a record corresponding to the character in the encoding dictionary database. When the judgment result is yes, sub-step 1205 is executed; when the judgment result is no, sub-step 1206 is executed;

[0049] Sub-step 1205, when the judgment result is yes, generating a matching string according to the string to be processed;

[0050] When it is determined that there is a record of the string to be processed in the encoding dictionary database, the string to be processed can be retained in the data cache area, and the above sub-steps 1203 to 1204 can be executed in a loop, and a new string to be processed is continued to be obtained from the original data as the second string to be processed, and the second string to be processed is added to the end of the string currently cached in the data cache area to obtain a new string to be processed, and then it is determined whether the new string to be processed has a matching substring in the encoding dictionary database.

[0051] Specifically, when it is determined that there is a record of the character string to be processed in the coding dictionary database, the communication system can move the data pointer (or the sliding time window) backward by at least one character position, and then the communication system can read the new character string to be processed according to the position pointed to by the pointer after the movement (or the area included in the sliding time window), and continue to execute sub-step 1204 until the communication system reads a character string that is not recorded in the coding dictionary database in the original data (for the convenience of description, the character string that is not recorded in the coding dictionary database is referred to as the unrecorded character string in the following text). At this time, in the data buffer area, the character string before the unrecorded character string is recorded in the coding dictionary database, so there is no need to repeatedly record these character strings in the coding dictionary database, and then the character string before the unrecorded character string can be used as a matching character string, and the matching character string is deleted from the data buffer area.

[0052] Sub-step 1206, when the judgment result is no, adding the character string to be processed to the encoding dictionary database;

[0053] When the judgment result is no, it means that the character string to be processed has no record in the coding dictionary database, and the character string to be processed needs to be stored in the coding dictionary database to complete the update and supplement of the coding dictionary database. After the character string to be processed is stored in the coding dictionary database, the communication system can clear the character string temporarily cached in the data cache area and cyclically execute the above sub-steps 1203 to 1204.

[0054] Sub-step 1207, complete scanning of all characters in the original data to obtain a coding dictionary corresponding to the original data.

[0055] By scanning all characters in the original data, the communication system can use the character strings contained in the original data to update and improve the preset coding dictionary database initialized by executing sub-step 1202, thereby obtaining a coding dictionary corresponding to the original data.

[0056] Step 13, searching for matching strings in each of the data blocks according to the coding dictionary, and determining matching positions and string lengths corresponding to the matching strings;

[0057] In one implementation, the communication information can search for repeated character strings in each data block according to a coding dictionary, use the repeated character strings as matching character strings, and record the position and character string length of each matching character string in the data block in the coding string.

[0058] Step 14, encoding each of the data blocks according to the encoding dictionary, the matching position and the string length, to obtain a first encoding string corresponding to the data block;

[0059] After determining the coding dictionary corresponding to the original data by executing step 12, the communication system may perform data encoding on each data block obtained by executing step 11 according to the coding dictionary to achieve the first coding compression of each data block.

[0060] Specifically, in one implementation, the communication system may encode the data blocks through the following sub-steps, including:

[0061] Sub-step 1401, for each character in each data block, searching in turn whether the character is recorded in the encoding dictionary;

[0062] Sub-step 1402, when it is determined that the character is recorded in the coding dictionary, the character is converted into a corresponding code according to the code in the coding dictionary, thereby completing the coding of the character in the data block;

[0063] Sub-step 1403, when it is determined that the character is not recorded in the encoding dictionary, the character can be converted into a binary number of a specific number of bits according to the encoding algorithm to complete the encoding of the character, and the character and the corresponding encoding of the character will be added to the encoding dictionary.

[0064] Sub-step 1404, replace the repeated character strings in the coded string with the matching positions and character string lengths of the repeated character strings (ie, matching character strings) to complete the first compression encoding of the data block and obtain the first coded character string corresponding to the data block.

[0065] Specifically, the first coded character string records the codes corresponding to the characters in the data block, and for repeated characters therein, the codes corresponding to the repeated characters are recorded only once, and the positions of the repeated characters and the length of the character string are recorded.

[0066] Step 15, re-encoding the matching position and the length of the string in the first encoded string to obtain a second encoded string corresponding to each data block;

[0067] In order to further improve the compression rate, in an embodiment of the present application, the matching position and the string length in the first encoded string obtained after the first encoding can be encoded twice to further reduce the data capacity.

[0068] In an embodiment of the present application, the communication system may use a range coding method (also referred to as interval coding) to perform secondary coding on the matching position and string length in the first coded string. Specifically, using the range coding method, a sufficiently large integer range and an estimate of the probability of the occurrence of a symbol or number may be given first, wherein the initial interval is easily divided into sub-intervals proportional to the probability of the symbol represented; then, the current interval is divided into sub-intervals corresponding to the probability of the next symbol to be encoded, and each symbol in the message may be encoded in this way.

[0069] In an embodiment of the present application, the matching position and the string length in the first encoded string can be re-encoded by the following method, including: determining a coding range based on the matching position and the string length; encoding the matching position and the string length based on the coding range to obtain a second encoded string corresponding to each of the data blocks.

[0070] In one implementation, the communication system may specifically determine the coding range according to the following method, including: constructing a probability model; determining the occurrence probability of the matching position and the string length according to the probability model; and determining the coding range according to the occurrence probability.

[0071] Specifically, the communication system can use the constructed probability model to count the occurrence frequencies of the matching positions and string lengths of the matching strings determined in the above step 14, and then determine the occurrence probabilities of the matching positions and string lengths corresponding to each matching string; according to the occurrence probabilities of the matching strings, the cumulative probability distribution function (CDF) can be used to calculate the cumulative probability of each matching position and string length, and the cumulative probability can be mapped to the range [0,1] to obtain the coding range.

[0072] It should be noted that the range coding method is a conventional technical means in the art, and the specific process of range coding will not be repeated in the embodiment of the present application.

[0073] Step 16, performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string, to obtain block compressed data corresponding to each of the data blocks;

[0074] Among them, predictive compression coding is a data compression technology that mainly uses the correlation in signal or image data for compression. This technology is based on a core idea: if there is a strong correlation between certain parts of the signal or image, then we can reduce the necessary amount of data by only storing or transmitting the difference between the actual data and the predicted data, thereby achieving data compression.

[0075] In one embodiment, the specific implementation method of step 16 may include: predicting the first encoded string and the second encoded string respectively according to the prediction model to obtain predicted strings; determining the prediction error according to the predicted strings and the actual strings; encoding the prediction error according to a preset compression algorithm to obtain block compressed data corresponding to each of the data blocks.

[0076] In one implementation, the prediction model may be constructed based on a Transformer model. The specific prediction model training method is as follows:

[0077] Sub-step 1601, constructing a prediction sample set according to historical transmitted data;

[0078] Sub-step 1602, extracting features from each sample data to be predicted in the prediction sample set to obtain feature data;

[0079] Sub-step 1603, inputting the extracted feature data into a pre-built Transformer model for forward propagation training to obtain a first prediction result;

[0080] Sub-step 1604, calculating the error between the first prediction result and the actual result according to a preset error loss function;

[0081] Sub-step 1605, back propagation is performed based on the error to update the weights of the Transformer model, and the above steps are repeated until the preset conditions are met to obtain a trained prediction model.

[0082] It should be noted here that the embodiments of the present application do not limit the specific construction and training method of the prediction model.

[0083] Step 17, combining the block compressed data to obtain compressed data corresponding to the original data.

[0084] According to a data compression method provided by an embodiment of the present application, before data transmission, the communication system can block the original data to be transmitted to obtain at least two data blocks, and encode each of the data blocks by constructing a coding dictionary to obtain a first coding string corresponding to the data block, and at the same time, according to the coding dictionary, find the matching string in each of the data blocks, and determine the matching position and string length corresponding to the matching string, and use the range coding method to perform secondary coding on the matching position and string length to obtain a second coding string, and finally, use the predictive compression coding method to compress the first coding string and the second string to obtain the final compressed data. According to the data compression method provided by the present scheme, since multiple coding methods (coding dictionary, range coding and predictive compression coding) are used to encode and compress the original data in turn, compared with the existing compression scheme that only uses a single coding method, the method provided by the present scheme can achieve high compression rate data compression, which improves the data compression efficiency on the one hand, and on the other hand, by transmitting compressed data, it greatly reduces the occupation of bandwidth resources in the data transmission process, and improves the data transmission efficiency.

[0085] In one implementation, the present application also provides a data decompression method to solve the problem of low compression efficiency in existing data compression schemes.

[0086] The specific implementation flow chart of the data decompression method provided in this application is as follows Figure 2 As shown, it mainly includes the following steps:

[0087] Step 21, decompress the compressed data to be decompressed, and obtain the encoding dictionary information, encoding string and encoding method information corresponding to the compressed data;

[0088] In one embodiment, for the received compressed data, the communication system can parse the compression header of the compressed data, and then obtain key parameters such as encoding dictionary information, encoding string and encoding method information corresponding to the compressed data for subsequent decompression process.

[0089] Step 22, constructing an empty decoding dictionary database according to the encoding dictionary information;

[0090] In an embodiment of the present application, the communication system can create an empty decoding dictionary database based on the encoding dictionary information in the compression header, and store the decompressed data during the decompression process through the decoding dictionary database for subsequent matching and prediction operations.

[0091] Step 23, decoding the encoded string according to the encoding method information to obtain a first decoded string and an undecoded string;

[0092] Step 24, decoding the undecoded string according to the range encoding method to obtain the matching position and the string length;

[0093] Step 25, performing secondary decoding on the first decoded string according to the encoding dictionary information, the matching position and the string length, to obtain the original data corresponding to the compressed data.

[0094] According to a data compression method provided by an embodiment of the present application, before data transmission, the communication system can block the original data to be transmitted to obtain at least two data blocks, and encode each of the data blocks by constructing a coding dictionary to obtain a first coding string corresponding to the data block, and at the same time, according to the coding dictionary, find the matching string in each of the data blocks, and determine the matching position and string length corresponding to the matching string, and use the range coding method to perform secondary coding on the matching position and string length to obtain a second coding string, and finally, use the predictive compression coding method to compress the first coding string and the second string to obtain the final compressed data. According to the data compression method provided by the present scheme, since multiple coding methods (coding dictionary, range coding and predictive compression coding) are used to encode and compress the original data in turn, compared with the existing compression scheme that only uses a single coding method, the method provided by the present scheme can achieve high compression rate data compression, which improves the data compression efficiency on the one hand, and on the other hand, by transmitting compressed data, it greatly reduces the occupation of bandwidth resources in the data transmission process, and improves the data transmission efficiency.

[0095] In one implementation, the present application also provides a data compression device to solve the problem of low compression efficiency in existing data compression solutions. Figure 3 As shown, it includes: a block division unit 31, a dictionary construction unit 32, a matching unit 33, an encoding unit 34 and a compression unit 35.

[0096] The block division unit 31 is used to perform block processing on the original data to obtain at least two data blocks;

[0097] A dictionary construction unit 32, used to construct a coding dictionary corresponding to the original data;

[0098] A matching unit 33, configured to search for a matching string in each of the data blocks according to the coding dictionary, and determine a matching position and a string length corresponding to the matching string;

[0099] The encoding unit 34 is used to encode each of the data blocks according to the encoding dictionary to obtain a first encoding string corresponding to the data block;

[0100] An encoding unit 34, used for encoding the matching position and the length of the string to obtain a second encoded string corresponding to each of the data blocks;

[0101] A compression unit 35 is used to perform predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to each of the data blocks;

[0102] The compression unit 35 is used to combine the block compressed data to obtain compressed data corresponding to the original data.

[0103] In one embodiment, the dictionary construction unit 32 is specifically used to: construct an empty original coding dictionary; initialize the original coding dictionary and determine the coding dictionary database corresponding to the original coding dictionary; scan the original data and obtain at least one character from the original data in turn to obtain a character string to be processed; determine whether the character string to be processed has a matching substring in the coding dictionary database; when the judgment result is yes, generate a matching character string according to the character string to be processed; when the judgment result is no, add the character string to be processed to the coding dictionary database; complete the scanning of all characters in the original data and obtain the coding dictionary corresponding to the original data.

[0104] In one implementation, the encoding unit 34 is specifically used to: determine an encoding range according to the matching position and the string length; encode the matching position and the string length according to the encoding range to obtain a second encoding string corresponding to each of the data blocks.

[0105] In one implementation, the encoding unit 34 is specifically used to: construct a probability model; determine the occurrence probability of the matching position and the string length according to the probability model; and determine the encoding range according to the occurrence probability.

[0106] In one embodiment, the compression unit 35 is specifically used to: predict the first encoded string and the second encoded string respectively according to the prediction model to obtain a predicted string; determine a prediction error according to the predicted string and the actual string; encode the prediction error according to a preset compression algorithm to obtain block compressed data corresponding to each data block.

[0107] By adopting a data compression device provided by an embodiment of the present application, before data transmission, the communication system can block the original data to be transmitted to obtain at least two data blocks, and encode each of the data blocks by constructing a coding dictionary to obtain a first coding string corresponding to the data block, and at the same time, according to the coding dictionary, find the matching string in each of the data blocks, and determine the matching position and string length corresponding to the matching string, and use the range coding method to perform secondary coding on the matching position and string length to obtain a second coding string, and finally, use the predictive compression coding method to compress the first coding string and the second string to obtain the final compressed data. By adopting the data compression method provided by the present scheme, since multiple coding methods (coding dictionary, range coding and predictive compression coding) are used to encode and compress the original data in turn, compared with the existing compression scheme that only uses a single coding method, the method provided by the present scheme can achieve data compression with a high compression rate, which improves the data compression efficiency on the one hand, and on the other hand, by transmitting compressed data, it greatly reduces the occupation of bandwidth resources in the data transmission process, thereby improving the data transmission efficiency.

[0108] In one implementation, the present application also provides a data decompression device to solve the problem of low compression efficiency in existing data decompression solutions. Figure 4 As shown, it includes: a decompression unit 41, a dictionary construction unit 42 and a decoding unit 43.

[0109] The decompression unit 41 is used to decompress the compressed data to be decompressed, and obtain the encoding dictionary information, encoding string and encoding method information corresponding to the compressed data;

[0110] A dictionary construction unit 42, configured to construct an empty decoding dictionary database according to the encoding dictionary information;

[0111] A decoding unit 43, configured to decode the encoded string according to the encoding mode information to obtain a first decoded string and an undecoded string;

[0112] A decoding unit 43, configured to decode the undecoded character string according to a range encoding method to obtain a matching position and a character string length;

[0113] The decoding unit 43 is used to perform secondary decoding on the first decoded string according to the encoding dictionary information, the matching position and the string length to obtain the original data corresponding to the compressed data.

[0114] By adopting a data decompression device provided by an embodiment of the present application, before data transmission, the communication system can block the original data to be transmitted to obtain at least two data blocks, and encode each of the data blocks by constructing a coding dictionary to obtain a first coding string corresponding to the data block, and at the same time, according to the coding dictionary, find the matching string in each of the data blocks, and determine the matching position and string length corresponding to the matching string, and use the range coding method to perform secondary coding on the matching position and string length to obtain a second coding string, and finally, use the predictive compression coding method to compress the first coding string and the second string to obtain the final compressed data. By adopting the data compression method provided by the scheme, since multiple coding methods (coding dictionary, range coding and predictive compression coding) are used to encode and compress the original data in turn, compared with the existing compression scheme that only uses a single coding method, the method provided by the scheme can achieve high compression rate data compression, which improves the data compression efficiency on the one hand, and on the other hand, by transmitting compressed data, it greatly reduces the occupation of bandwidth resources in the data transmission process, and improves the data transmission efficiency.

[0115] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0116] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0117] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0118] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data compression device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: block processing of the original data to obtain at least two data blocks; constructing a coding dictionary corresponding to the original data; searching for matching strings in each of the data blocks according to the coding dictionary, and determining the matching position and string length corresponding to the matching string; encoding each of the data blocks according to the coding dictionary, the matching position and the string length to obtain a first coding string corresponding to the data block; encoding the matching position and the string length to obtain a second coding string corresponding to each of the data blocks; performing predictive compression coding on each of the data blocks according to the first coding string and the second coding string to obtain block compressed data corresponding to each of the data blocks; combining the block compressed data to obtain compressed data corresponding to the original data.

[0119] The above application Figure 5The method performed by the data compression electronic device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0120] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute Figure 1 The method of the embodiment shown is specifically used to perform the following operations:

[0122] The original data is processed into blocks to obtain at least two data blocks; a coding dictionary corresponding to the original data is constructed; according to the coding dictionary, matching strings in each of the data blocks are searched respectively, and matching positions and string lengths corresponding to the matching strings are determined; according to the coding dictionary, the matching positions and the string lengths, each of the data blocks is encoded respectively to obtain a first coding string corresponding to the data block; the matching position and the string length are encoded to obtain a second coding string corresponding to each of the data blocks; according to the first coding string and the second coding string, each of the data blocks is predictively compressed and encoded respectively to obtain block compressed data corresponding to each of the data blocks; the block compressed data are combined to obtain compressed data corresponding to the original data.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0129] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0131] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data compression method, characterized in that: include: The original data is processed into blocks to obtain at least two data blocks; Constructing a coding dictionary corresponding to the original data; According to the coding dictionary, searching for matching strings in each of the data blocks respectively, and determining matching positions and string lengths corresponding to the matching strings; According to the coding dictionary, the matching position and the string length, each of the data blocks is encoded respectively to obtain a first coding string corresponding to the data block; Encoding the matching position and the character string length to obtain a second encoded character string corresponding to each of the data blocks; According to the first encoding string and the second encoding string, predictive compression encoding is performed on each of the data blocks respectively to obtain block compressed data corresponding to each of the data blocks; The block compressed data are combined to obtain compressed data corresponding to the original data.

2. The method according to claim 1, characterized in that The constructing of a coding dictionary corresponding to the original data specifically includes: Construct an empty original encoding dictionary; Initializing the original coding dictionary and determining a coding dictionary database corresponding to the original coding dictionary; Scanning the original data, acquiring at least one character from the original data in sequence, and obtaining a character string to be processed; Determine whether the character string to be processed has a matching substring in the encoding dictionary database; When the judgment result is yes, generating a matching string according to the string to be processed; When the judgment result is no, adding the character string to be processed to the encoding dictionary database; Complete scanning of all characters in the original data to obtain a coding dictionary corresponding to the original data.

3. The method according to claim 1, characterized in that: The encoding of the matching position and the string length to obtain a second encoded string corresponding to each of the data blocks specifically includes: Determine a coding range according to the matching position and the length of the character string; The matching position and the character string length are encoded according to the encoding range to obtain a second encoded character string corresponding to each of the data blocks.

4. The method according to claim 3, characterized in that: Determining the encoding range according to the matching position and the string length specifically includes: Build probability models; Determining the occurrence probability of the matching position and the length of the character string according to the probability model; The coding range is determined according to the occurrence probability.

5. The method according to claim 1, characterized in that The predictive compression encoding is performed on each of the data blocks according to the first encoding string and the second encoding string to obtain the block compressed data corresponding to each of the data blocks, specifically including: According to the prediction model, predict the first encoded string and the second encoded string respectively to obtain predicted strings; Determining a prediction error based on the predicted character string and the actual character string; The prediction error is encoded according to a preset compression algorithm to obtain block compressed data corresponding to each data block.

6. A data decompression method, characterized in that: include: Decompressing the compressed data to be decompressed, and obtaining encoding dictionary information, encoding character string, and encoding method information corresponding to the compressed data; Constructing an empty decoding dictionary database according to the encoding dictionary information; Decoding the encoded string according to the encoding method information to obtain a first decoded string and an undecoded string; Decoding the undecoded string according to the range encoding method to obtain the matching position and the string length; The first decoded string is decoded again according to the encoding dictionary information, the matching position and the string length to obtain the original data corresponding to the compressed data.

7. A data compression device, characterized in that: include: A block division unit, used for performing block processing on the original data to obtain at least two data blocks; A dictionary construction unit, used to construct a coding dictionary corresponding to the original data; A matching unit, used to search for a matching string in each of the data blocks according to the coding dictionary, and determine a matching position and a string length corresponding to the matching string; An encoding unit, used for encoding each of the data blocks according to the encoding dictionary to obtain a first encoding character string corresponding to the data block; An encoding unit, used for encoding the matching position and the length of the string to obtain a second encoded string corresponding to each of the data blocks; A compression unit, configured to perform predictive compression coding on each of the data blocks according to the first coding string and the second coding string, to obtain block compressed data corresponding to each of the data blocks; The compression unit is used to combine the block compressed data to obtain compressed data corresponding to the original data.

8. A data compression device, comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: The original data is processed into blocks to obtain at least two data blocks; Constructing a coding dictionary corresponding to the original data; According to the coding dictionary, searching for matching strings in each of the data blocks respectively, and determining matching positions and string lengths corresponding to the matching strings; According to the coding dictionary, the matching position and the string length, each of the data blocks is encoded respectively to obtain a first coding string corresponding to the data block; Encoding the matching position and the character string length to obtain a second encoded character string corresponding to each of the data blocks; According to the first encoding string and the second encoding string, predictive compression encoding is performed on each of the data blocks respectively to obtain block compressed data corresponding to each of the data blocks; The block compressed data are combined to obtain compressed data corresponding to the original data.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enable the electronic device to execute the data compression method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the data compression method as claimed in any one of claims 1 to 6.

Images