Data compression and decompression method and unit, mobile terminal, cloud computing center, communication method and system

Abstract

The invention discloses a data compression and decompression method, a data compression and decompression unit, a mobile terminal, a cloud computing center, a communication method and a communication system, relates to the field of satellite communication, and is used for relieving the satellite transmission pressure. The data compression method is configured in the mobile terminal, LZ77 coding is carried out on data blocks with the length meeting the requirement after segmentation to obtain Literals and a triple sequence, Huffman compression is carried out on the Literals, and FSE compression is carried out on a Huffman compression packet; and performing FSE compression on the triple sequence. And configuring a corresponding data decompression method in the cloud computing center. And the plurality of mobile terminals respectively transmit the compressed data to the cloud computing center for decompression application through satellites. According to the method, the compression rate of the short character string data is improved, the stable compression rate is achieved, the transmission pressure of a satellite can be obviously relieved, and data of a mobile communication coverage blind area can be obtained in time.

Description

Technical Field

[0001] The present invention relates to the field of satellite communication, in particular to a data compression method and unit, a data decompression method and unit, a mobile terminal equipped with a data compression unit, a cloud computing center equipped with a data decompression unit, a communication method based on the data compression method and the data decompression method, and a communication system based on the mobile terminal and the cloud computing center. Background Art

[0002] In terms of satellite network transmission services, current satellite bandwidth resources are limited and cannot support communication and data transmission needs in various fields around the world.

[0003] Compression of transmitted data can alleviate the current network transmission pressure to a certain extent. In satellite communications, the compression performance of conventional string compression algorithms is not high enough, and the system resource consumption is relatively large. When low-performance terminals face large-scale IoT short message data, they appear to be insufficient. Summary of the invention

[0004] The purpose of the present invention is to provide a data compression and decompression method, unit, mobile terminal, cloud computing center, communication method and system to address all or part of the above-mentioned problems, thereby improving the compression performance by improving the data compression algorithm, thereby alleviating the pressure of satellite transmission, and dispersing the data compression tasks through distributed networking through mobile terminals, alleviating the pressure of central equipment, and improving networking flexibility.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A data compression method, wherein the data includes at least one data block, and the at least one data block is compressed by the following method:

[0007] Performing a first encoding on the data block to obtain first data and second data both including the maximum number of occurrences of the indicating character;

[0008] When the maximum number of occurrences of a character in the first data reaches a second threshold, performing a first compression of the first data according to an indication of the maximum number of occurrences of a character in the first data;

[0009] When the maximum number of occurrences of a character in the second data reaches a second threshold, performing a second compression on the second data according to an indication of the maximum number of occurrences of a character in the second data;

[0010] A compression result of the data block is obtained.

[0011] Accordingly, the present invention also provides a data decompression method for decompressing compressed data, the method comprising:

[0012] Obtaining an identifier of the data, and determining whether the data is compressed according to the identifier;

[0013] If the data is compressed, then according to the indication of the identifier, the compression result of each compressed data block of the data is decompressed using the following methods:

[0014] According to the indication of the identifier, decompress the second data compressed in the compression result of the data block, or directly obtain the compressed second data;

[0015] According to the indication of the identifier, decompressing the compressed first data in the compression result of the data block, or directly obtaining the compressed first data;

[0016] The data obtained in the first two steps are subjected to a first decoding corresponding to the first encoding to obtain a decompression of the compression result of the data block.

[0017] The present invention further provides a data compression unit, wherein the data includes at least one data block, and the data compression unit includes:

[0018] The encoding module is configured to: perform a first encoding on at least one of the data blocks to obtain first data and second data both including the maximum number of occurrences of the indicating character;

[0019] A first compression module is configured to: when the maximum number of occurrences of a character in the first data reaches a second threshold, perform a first compression on the first data according to an indication of the maximum number of occurrences of a character in the first data;

[0020] A second compression module is configured to: when the maximum number of times a character appears in the second data reaches a second threshold, perform a second compression on the second data according to an indication of the maximum number of times a character appears in the second data;

[0021] The output module is configured to: obtain the output results of the first compression module and the second compression module, and output the compression result of the data block.

[0022] The present invention also provides a data decompression unit for decompressing compressed data, the data decompression unit comprising:

[0023] The identifier acquisition module is configured to: obtain an identifier of the data, and determine whether the data is compressed according to the identifier; if the data is determined to be compressed, enable the first decompression module, the second decompression module and the decoding module to decompress the compression result of each compressed data block of the data;

[0024] A first decoding module is configured to: perform first compression decompression of the compressed first data in the compression result of the data block according to the instruction of the identifier, or directly obtain the compressed first data;

[0025] The second decoding module is configured to: perform decompression of the second compressed data in the compression result of the data block according to the instruction of the identifier, or directly obtain the compressed second data;

[0026] The decoding module is configured to: perform a first decoding corresponding to the first encoding on the data obtained by the first decoding module and the second decoding module to decompress the compression result of the data block.

[0027] The present invention also provides a mobile terminal, which is equipped with the above data compression unit.

[0028] The present invention also provides a cloud computing center, which is equipped with the above-mentioned data decompression unit.

[0029] The present invention also provides a communication system, which includes at least one of the above-mentioned mobile terminals and the above-mentioned cloud computing center, each of the mobile terminals is respectively communicated with the cloud computing center, each of the mobile terminals compresses the acquired data using a data compression unit and sends the data to the cloud computing center, and the cloud computing center decompresses the received data using a data decompression unit.

[0030] The present invention also provides a communication method, comprising:

[0031] Collect data;

[0032] Compressing the collected data using the above data compression method;

[0033] Send compressed data;

[0034] Receive compressed data;

[0035] The compressed data is decompressed using the above data decompression method.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] 1. The data compression scheme of the present invention has significantly improved the compression rate and compression speed of short character strings compared with existing compression algorithms, has strong compression stability, effectively avoids the situation of negative compression rate, and avoids the disadvantage of increased data volume. In particular, when applied to the communication system of the present invention, it can significantly reduce the transmission pressure of the satellite.

[0038] 2. The present invention utilizes the edge acquisition system to aggregate data from multiple sensors and then reports it. On the one hand, it can realize distributed management of sensors and reduce the number of acquisition devices. On the other hand, the edge acquisition system is configured with dual strategy control of periodic reporting and reporting according to capacity threshold, ensuring data integrity and transmission stability.

[0039] 3. The present invention utilizes mobile terminals as devices for data compression and transmission. One mobile terminal can manage the data of multiple edge acquisition systems, reducing the number of satellite access devices. In addition, the mobile terminal transmits data to the cloud computing center via satellite, thereby achieving coverage of data collection in areas without mobile communication networks, which can reduce the workload required for manual collection in such areas and improve the timeliness of data collection. At the same time, in addition to data compression, the collected data can also be aggregated, sorted, cleaned, reconstructed, and other pre-processed with the help of mobile terminals to remove polluted data and invalid redundant data, thereby minimizing the consumption of transmission resources by the amount of data and saving communication resources and communication time. In addition, by using mobile terminals as edge devices to form an edge network, new mobile terminals can be added at any time as needed, or mobile terminals can be exited from the system to achieve flexible networking and improve system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0041] Figure 1 is a flow chart of an embodiment of a data compression method.

[0042] Figure 2 is a flow chart of a data compression method in another embodiment.

[0043] Figure 3 It is a flow chart of an embodiment of the operation of a mobile terminal.

[0044] Figure 4 is an architectural diagram of an embodiment of a communication network.

[0045] Figure 5 It is a flow chart of an embodiment of the operation of the edge acquisition system.

[0046] Figure 6 and Figure 7 These are comparison charts of compression ratios of various compression algorithms.

[0047] Figure 8 It is a comparison diagram of memory usage of the data compression / decompression method of the present invention in one embodiment.

[0048] Fig. 9 This is a comparison chart of memory usage for compression / decompression using the zlib algorithm. DETAILED DESCRIPTION

[0049] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0050] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0051] Example 1

[0052] This embodiment introduces a data compression method for compressing short character string data transmitted via a satellite.

[0053] Here, sensor data is taken as an example. Many sensors (including temperature, humidity, air quality parameters and other sensor devices or monitoring devices) are installed at the remote end. It is assumed that the sensor data collected by these sensors needs to be compressed.

[0054] The amount of sensor data collected by many sensors is large and cannot be compressed directly. Therefore, the data to be compressed is usually processed in blocks, that is, the sensor data to be compressed is cut into multiple data blocks, and then at least one data block of the sensor data is compressed. That is, all data blocks of the sensor data can be compressed, or only some of the data blocks can be compressed.

[0055] For the data blocks to be compressed, the following methods are used for compression:

[0056] 1) Performing a first encoding on the database to obtain first data and second data, both of which contain parameters indicating the maximum number of occurrences of a character, which may be direct statistical parameters or character representation parameters that can be statistically analyzed, and then statistically analyzed during calculation.

[0057] 2) Performing a first compression of the first data at least according to the indication of the maximum number of occurrences of the characters in the first data, or not compressing the first data. That is, the maximum number of occurrences of the characters in the first data is used as a parameter or one of the parameters for determining whether to perform a first compression on the first data. This step obtains the compressed first data.

[0058] 3) performing a second compression on the second data at least according to the indication of the maximum number of occurrences of the character in the second data, or not compressing the second data. This step obtains the compressed second data.

[0059] 4) After the operations of steps 2) and 3) above, the compression result of the data block is obtained.

[0060] The compression results of each data block are written into the data stream in turn, thus obtaining compressed sensor data.

[0061] Obviously, the first encoding mentioned above is not an arbitrary encoding method, and it is necessary to obtain the first data and the second data that meet the above requirements. In some specific embodiments, the first encoding adopts LZ77 encoding, which encodes the data block to obtain Literals (i.e., character count) and a triple sequence <character position offset, character length, current character (next character)>, Literals is equivalent to the first data in the above method, and the triple sequence is equivalent to the second data. Both contain parameters that directly or indirectly indicate the maximum number of times a character appears. The subsequent introduction of this embodiment will be illustrated using the example of LZ77 encoding.

[0062] In addition, in this embodiment, not all data blocks need to be LZ77 encoded. Before LZ77 encoding is performed, it is first determined whether the data block needs to be LZ77 encoded.

[0063] In some specific embodiments, whether to perform LZ77 encoding on the data block is determined based on whether the length of the data block reaches a first threshold. For example, when the length of the data block does not reach 4kbyte (or other thresholds, which can be selected by yourself), the data block is not encoded, and obviously no subsequent compression is performed.

[0064] The encoded Literals are further subjected to a first compression process. In this embodiment, the Huffman compression algorithm is used to compress Literals during the first compression process. Similarly, before deciding to use the Huffman compression algorithm to compress Literals, it is first decided whether to perform Huffman compression on Literals based on whether the maximum number of times a character in Literals appears reaches a second threshold (such as 3 times, which can be changed to other thresholds at will), that is, based on the maximum number of times a character in Literals appears, it is decided whether to perform Huffman compression on Literals or not.

[0065] For the first compression process of Literals, in the Huffman compression algorithm, other compression steps are nested to perform a third compression process on the intermediate result of the first compression, and the intermediate result indicates the mapping relationship between the characters before and after the first compression. According to the example of using the Huffman compression algorithm, the intermediate result is the Huffman compression table.

[0066] In some specific embodiments, after deciding to perform Huffman compression on Literals, it is necessary to first determine whether to perform a third compression process on the Huffman compression table. Here, the FSE compression algorithm is used as an example of the algorithm for the third compression process, that is, first determine whether to use the FSE compression algorithm to compress the Huffman compression table.

[0067] In this embodiment, the judgment of whether to use the FSE compression algorithm to compress the Huffman compression table is implemented by using a trial mechanism, that is, first try to compress the Huffman compression table with the FSE compression algorithm, and then see whether the compression effect meets the requirements, so as to decide whether to use the compression result:

[0068] First, the Huffman compression table is pre-compressed using the FSE compression algorithm;

[0069] Calculate the performance of pre-compression, such as the length of the compressed stream (i.e., the compressed data) after pre-compression; based on the set first condition, evaluate whether the pre-compression performance meets the first condition, so as to decide whether to use the FSE compression algorithm to compress the Huffman compression table. Assume that the first condition is: the length of the compressed stream is less than half of the statistical maximum valid symbol value. Then, when the pre-compressed compressed stream meets the first condition, the result of pre-compression is adopted (the Huffman compression table is no longer re-compressed), that is, it is determined to use the FSE compression algorithm to compress the Huffman compression table. If the first condition is not met, the FSE compression algorithm is not used to compress the Huffman compression table. In the first compression, only the Huffman compression algorithm is used to compress Literals.

[0070] After determining whether to use the FSE compression algorithm to compress the Huffman compression table, it is necessary to write the Huffman compressed data stream and the Huffman compression table into the compression stream to complete the first compression process of Literals.

[0071] For a triple sequence, a second compression is performed on the triple sequence according to an indication of the maximum number of occurrences of a character therein, or no compression is performed. In some specific embodiments, similar to the above determination of whether to perform Huffman compression, whether to perform a second compression on the triple sequence is determined based on whether the maximum number of occurrences of a character in the triple sequence reaches a second threshold (such as 3 times, which can be changed to other thresholds).

[0072] The second compression here, in this embodiment, is also performed using the FSE compression algorithm, that is, whether to use the FSE compression algorithm to compress the triple sequence is determined based on whether the maximum number of occurrences of characters in the triple sequence reaches the second threshold. If FSE compression is used, the triple sequence is compressed and written into the compressed stream, otherwise the triple sequence is directly written into the compressed stream.

[0073] After the above compression processing of Literals and triple sequences is completed, the part written to the compressed stream is the compression result of the data block. Repeat the compression process for each data block that needs to be compressed, that is, the compression processing of all data blocks that need to be compressed is realized, and finally the sensor data is compressed.

[0074] According to the above introduction, if Figure 1 As shown, the complete compression process of the collected sensor data includes the following steps:

[0075] 1) Initialization.

[0076] First, the metadata (ie, the data to be compressed) is cut into multiple data blocks, each of which is at most 128KB, and each data block is read into the cache in sequence.

[0077] 2) Determine whether LZ77 encoding is required.

[0078] Determine whether the length of each data block is less than 4kbyte (adjustable). If it is less than 4kbyte, LZ77 encoding is not required, and jump to step 5). Otherwise, select relevant parameters according to LZ77 encoding, and obtain the corresponding two outputs: Literals and triple sequences. Literals is the character count, and the triple sequence is <character position offset, character length, current character (next character)>.

[0079] 3) Determine whether Huffman compression of Literals is required.

[0080] Based on the judgment of parameters such as the maximum number of times characters appear in Literals, it is determined whether the Huffman compression algorithm needs to be used to compress Literals. The judgment threshold is 3 by default, that is, it is necessary when the maximum number of times a character appears reaches 3. This threshold is adjustable. If it is determined to be necessary, Huffman compression is performed on Literals, and the Huffman compression table and the compressed data stream are calculated and generated. If not required, jump to step 5). In addition, if Huffman compression is required, first try to compress the Huffman compression table with the FSE compression algorithm. When the compressed stream length of the Huffman compression table after FSE compression is less than half of the maximum valid symbol value counted, the FSE compression algorithm is used to compress the Huffman compression table. Otherwise, the compression result of the FSE compression algorithm is not used.

[0081] 4) Determine whether FSE compression is required.

[0082] According to the maximum number of times a character appears in the triple sequence after sequence mapping and other related parameters, it is determined whether the triple sequence needs to be compressed by FSE. The judgment threshold is 3 by default, that is, it is required when the maximum number of times a character appears reaches 3. This threshold is adjustable. If it is not required, jump to step 5). Otherwise, the triple sequence is compressed by FSE.

[0083] 5) Construct a compressed block Block.

[0084] The results generated in the above steps are packaged to construct a Block. Each Block includes a Blockheader to distinguish different compressed blocks.

[0085] 6) Determine whether the current data block is the last one.

[0086] If the currently compressed data block is the last data block of the metadata, the compression ends; otherwise, the process returns to step 2) to continue compressing the next data block.

[0087] In the above data compression method, the data compression process is explained by taking the sensor data as metadata as an example. In fact, after compressing the metadata, this embodiment also determines whether it is necessary to retain the compression result, that is, if negative compression occurs, there is no need to use the compressed data, and the metadata can be sent directly. In this regard, on the basis of the above data compression method, it is also determined whether the length of the compressed sensor data is reduced. If so, an identifier indicating which compression processes have been passed is added to the compressed sensor data. In addition to indicating whether each data block has been compressed, the identifier also indicates whether each data block has been LZ77 encoded, Huffman compressed, FSE compressed, etc.; if not, an uncompressed identifier is added to the sensor data, and the sensor data before compression is retained for transmission. The role of the identifier is to inform the receiving end that no decompression processing is required, and the received data can be used directly. In some specific embodiments, such as Figure 2 As shown, a frame header is added to the compressed data packet, and identifiers 1-9 indicating which compression processes have been passed are written in the frame header. A frame header is also added to the uncompressed original data, and the uncompressed identifier 0 is written in the frame header.

[0088] In addition, for the compression of sensor data, the sensor data needs to meet certain data types, that is, before compressing the sensor data, it is determined whether the sensor data meets the compressed data type. If so, the sensor data is compressed, otherwise an error is reported or other responses are made. Figure 2As shown, in some specific embodiments, the sensor data is required to be byte type data. Only when the sensor data is determined to be byte type data, the sensor data is compressed by the data compression method described above, otherwise an error is reported. In this regard, the device configured with the data compression algorithm is required to have a module circuit that converts the format of the data to be compressed into a byte type. For example, assuming that the data compression algorithm is configured in a mobile terminal (such as a mobile phone), the mobile terminal also needs to be configured with a unit (called a format conversion unit) that converts the format of the acquired data into a byte type.

[0089] Example 2

[0090] This embodiment introduces a data compression unit. The data to be compressed includes at least one data block. The data blocks may be divided in the data compression unit, or may have been divided into multiple data blocks when input into the data compression unit.

[0091] The data compression unit includes an encoding module, a first compression module, a second compression module and an output module. If the metadata needs to be segmented in the data compression unit, the data compression unit also includes a data block segmentation module. If it is for already segmented data blocks, the data block segmentation module is not needed. Wherein:

[0092] The data block segmentation module is configured to: segment the input metadata into multiple data blocks, each data block is segmented according to the configured maximum length limit, and read each data block into the cache to be called by subsequent modules.

[0093] The encoding module is configured to: perform a first encoding on at least one data block to obtain first data and second data both containing the maximum number of occurrences of the indicating character.

[0094] The first compression module is configured to: perform a first compression on the first data at least according to an indication of a maximum number of occurrences of a character in the first data, or perform no compression.

[0095] The second compression module is configured to: perform a second compression on the second data at least according to an indication of a maximum number of occurrences of a character in the second data, or perform no compression.

[0096] The output module is configured to: obtain the output results of the first compression module and the second compression module, and output the compression results of the data block.

[0097] As in the example of Embodiment 1, the first encoding algorithm configured in the encoding module is the LZ77 encoding algorithm, and the first data and the second data output by the encoding module are Literals and a triple sequence, respectively. Correspondingly, the first compression algorithm configured in the first compression module includes the Huffman compression algorithm and the FSE compression algorithm, and the second compression algorithm configured in the second compression module is the FSE compression algorithm.

[0098] Specifically, the operation configured in the encoding module determines whether the data block needs to be LZ77 encoded before LZ77 encoding the database. Specifically, whether to perform LZ77 encoding on the data block can be determined based on whether the length of the data block reaches a first threshold. For example, when the length of the data block does not reach 4kbyte (or other thresholds, which can be selected by the user), the data block is not encoded, and obviously no subsequent compression is performed.

[0099] The first compression operation configured in the first compression module, in some specific embodiments, includes compressing Literals using the Huffman compression algorithm. In addition, before deciding to compress Literals using the Huffman compression algorithm, it is first decided whether to perform Huffman compression on Literals based on whether the maximum number of times a character in Literals appears reaches a second threshold (such as 3 times, which can be changed to other thresholds at will), that is, based on the maximum number of times a character in Literals appears, it is decided whether to perform Huffman compression on Literals or not.

[0100] In addition, the first compression operation configured in the first compression module also has other compression steps embedded in the Huffman compression algorithm to perform a third compression process on the intermediate result of the first compression, and the intermediate result indicates the mapping relationship between the characters before and after the first compression, that is, the Huffman compression table. Taking the FSE compression algorithm as an example of the algorithm for the third compression process, after deciding to perform Huffman compression on Literals, it is first determined whether to use the FSE compression algorithm to compress the Huffman compression table.

[0101] In the first compression module, the configured trial mechanism is used to determine whether to use the FSE compression algorithm to compress the Huffman compression table. The trial mechanism first tries to compress the Huffman compression table with the FSE compression algorithm, and then sees whether the compression effect meets the requirements, thereby deciding whether to use the compression result:

[0102] First, the Huffman compression table is pre-compressed using the FSE compression algorithm;

[0103] Calculate the performance of pre-compression, such as the length of the compressed stream (i.e., the compressed data) after pre-compression; based on the set first condition, evaluate whether the pre-compression performance meets the first condition, so as to decide whether to use the FSE compression algorithm to compress the Huffman compression table. Assume that the first condition is: the length of the compressed stream is less than half of the statistical maximum valid symbol value. Then, when the pre-compressed compressed stream meets the first condition, the result of pre-compression is adopted (the Huffman compression table is no longer re-compressed), that is, it is determined to use the FSE compression algorithm to compress the Huffman compression table. If the first condition is not met, the FSE compression algorithm is not used to compress the Huffman compression table. In the first compression, only the Huffman compression algorithm is used to compress Literals.

[0104] After determining whether to use the FSE compression algorithm to compress the Huffman compression table, the first compression module writes the Huffman compressed data stream and the Huffman compression table into the compression stream, completing the first compression process of Literals.

[0105] The second compression operation configured in the second compression module determines whether to perform a second compression on the triple sequence based on whether the maximum number of occurrences of characters in the triple sequence reaches a second threshold value (such as 3 times, which can be changed to other threshold values). In some specific embodiments, the second compression is configured to use an FSE compression algorithm. If FSE compression is used, the triple sequence is compressed and written into the compressed stream, otherwise the triple sequence is directly written into the compressed stream.

[0106] The output module sequentially inputs the compression results of each data block of the sensor data that needs to be compressed into the compression stream, that is, the sensor data is compressed. The configuration of the corresponding module of the data compression unit can refer to the features executed by the corresponding steps in Example 1.

[0107] Example 3

[0108] This embodiment introduces a data decompression method for decompressing sensor data compressed by the above method or unit. The compressed sensor data carries an identifier indicating whether it is compressed or which compression processes it has undergone.

[0109] First, obtain the identifier of the (compressed) data, and determine whether the data is compressed based on the identifier. If it is not compressed, directly use the received sensor data as the decompressed sensor data. If it is compressed, reversely decompress the compression result of each data block according to the indication of the identifier to restore the original data. The process includes:

[0110] 1) decompressing the compressed second data in the compression result of the data block according to the instruction of the identifier, or directly obtaining the compressed second data; and

[0111] 2) performing a first compression decompression on the first data compressed in the compression result of the data block according to the instruction of the identifier, or directly obtaining the compressed first data;

[0112] 3) Then, the data obtained in the previous steps 1) and 2) are subjected to a first decoding corresponding to the first encoding to obtain a decompression result of the compression result of the data block.

[0113] According to the data encoding method in Example 1, in the data decomposition method of this embodiment, the compression result of the obtained data block includes Literals, or a data stream compressed by Huffman, a Huffman compression table; and a triple sequence compressed by / without FSE, the first two of which are the first data compressed in the compression result, and the last one is the second data compressed in the compression result.

[0114] In the above step 1), if the identifier indicates that the triple sequence has been FSE compressed, step 1) performs FSE decompression on the FSE compressed triple sequence; if the identifier indicates that the triple sequence has not been FSE compressed, step 2) directly outputs the triple sequence.

[0115] In the above step 2), if the identifier indicates that Literals has been Huffman compressed, the compressed first data includes the Huffman compressed data stream and the Huffman compression table.

[0116] (1) If the identifier indicates that the Huffman compression table is FSE compressed, then step 2) first decompress the Huffman compression table by FSE, and then decompress the data stream according to the decompressed Huffman compression table to recover the Literals;

[0117] (2) If the identifier indicates that the data stream has not been FSE compressed, step 2) directly decompresses the data stream using the received Huffman compression table;

[0118] If the identifier refers to Literals that have not been Huffman compressed, step 2) directly outputs Literals.

[0119] In the above step 3), the decompressed Literals and triplet sequences are subjected to LZ77 decoding to restore the data blocks.

[0120] Each data is decompressed according to the above process to complete the decompression of the compressed sensor data.

[0121] Example 4

[0122] This embodiment introduces a data decompression unit for decompressing the sensor data compressed by the above method or unit. The compressed sensor data carries an identifier indicating whether it is compressed or which compression processes it has undergone.

[0123] The data decompression unit includes:

[0124] The identifier acquisition module is configured to: obtain an identifier of the compressed sensor data, and determine whether the data is compressed based on the identifier; if the sensor data is determined to be compressed, enable the first decompression module, the second decompression module and the decoding module to decompress the compression result of each data block of the obtained compressed sensor data.

[0125] The first decoding module is configured to: perform first compression decompression on the compressed first data in the compression result of the data block according to the instruction of the identifier, or directly obtain the compressed first data.

[0126] The second decoding module is configured to: perform second compression decompression on the second data compressed in the compression result of the data block according to the indication of the identifier, or directly obtain the compressed second data.

[0127] The decoding module is configured to: perform a first decoding corresponding to the first encoding on the data obtained by the first decoding module and the second decoding module to obtain a decompression result of the data block.

[0128] According to the above embodiments, the compression result of the obtained data block includes Literals, or a data stream after Huffman compression, a Huffman compression table; and a triple sequence compressed / uncompressed by FSE, the first two of which are the first data compressed in the compression result, and the last one is the second data compressed in the compression result.

[0129] In the first decompression module, if the identifier indicates that the Literals have been Huffman compressed, the compressed first data includes the Huffman compressed data stream and the Huffman compression table.

[0130] (1) If the identifier indicates that the Huffman compression table is FSE compressed, the operation configured in the first decompression module first performs FSE decompression on the Huffman compression table, and then decompresses the data stream according to the decompressed Huffman compression table to recover Literals;

[0131] (2) if the identifier indicates that the data stream has not been FSE compressed, the operation configured in the first decompression module directly uses the received Huffman compression table to decompress the data stream;

[0132] If the identifier indicates that Literals have not been Huffman compressed, the operation configured in the first decompression module directly outputs Literals.

[0133] In the above-mentioned second decompression module, if the identifier indicates that the triple sequence has been FSE compressed, the operation configured in the second decompression module performs FSE decompression on the triple sequence compressed by FSE; if the identifier indicates that the triple sequence has not been FSE compressed, the operation configured in the second decompression module directly outputs the triple sequence.

[0134] The operations configured in the above decoding module perform LZ77 decoding on the decompressed Literals and triplet sequences to recover the data blocks.

[0135] Example 5

[0136] This embodiment introduces a communication system, such as Figure 3 As shown, the system includes at least one mobile terminal and a cloud computing center, and a plurality of mobile terminals are respectively connected to the cloud computing center for communication. The mobile terminals can freely access the communication system or exit from the communication system.

[0137] In some specific embodiments, the mobile terminals are each equipped with a data compression unit, and the cloud computing center is equipped with a data decompression unit. Each mobile terminal compresses the acquired data using the data compression unit and sends it to the cloud computing center, and the cloud computing center decompresses the received data using the data decompression unit.

[0138] In addition, the mobile terminal in this embodiment can be a portable intelligent terminal such as a mobile phone, a PDA, etc., which has a certain data processing capability. In addition to being configured with a data compression unit, it also includes a receiving unit for receiving sensor data, a preprocessing unit for preprocessing the sensor data, a format conversion unit for converting the sensor data into a data type that satisfies compression, and a sending unit for sending compressed data. As a feasible implementation method, the receiving unit, the preprocessing unit, the format conversion unit, the data compression unit and the sending unit are connected in sequence, that is, the operations configured by each unit are completed in sequence. Obviously, the possible cyclic nested connection between some units on this basis is not excluded.

[0139] The preprocessing unit is configured to perform one or more of the following processes: aggregation, sorting, cleaning, and reconstruction on the sensor data received by the receiving unit. Aggregation or sorting includes extracting key data to remove invalid redundant data. Data cleaning can filter out polluted data. The format conversion unit converts the preprocessed sensor data into a data type such as byte type that satisfies data compression.

[0140] like Figure 4As shown, after the mobile terminal uses the preprocessing unit to clean, integrate, extract key data and other preprocessing of the sensor data, it first determines whether the sensor data needs to be compressed according to current needs. If necessary, the format conversion unit is used to convert the sensor data into a byte type, and then the data compression unit is used to compress it to obtain a byte type compression package. Finally, the sending unit is used to transmit the compressed sensor data to the cloud computing center.

[0141] In some specific embodiments, the mobile terminal and the cloud computing center are connected by communication via a satellite, so that timely sensor data can be acquired in areas without mobile network coverage.

[0142] like Figure 3 As shown in the figure, the sensor data acquired by each mobile terminal is reported through the edge collection system. A mobile terminal can be connected to multiple edge collection systems, and the edge collection system interacts with the mobile terminal through LAN, WIFI, or cellular mobile network. An edge collection system can connect to multiple sensors, manage multiple connected sensors at the same time, and control the collection frequency of each sensor. Each sensor is responsible for sending the collected sensor data to the connected edge collection system for aggregation, and the edge collection system is responsible for aggregating the sensor data collected by the sensor and transmitting it to the connected mobile terminal.

[0143] Taking into account the timeliness requirements of on-site data reporting and the resource consumption of compressed data on mobile terminals, dual strategy control is implemented on the rules for reporting sensor data by the edge collection system. When the configured upload cycle or upload capacity threshold is met, the edge collection system will transmit the summarized sensor data to the mobile terminal.

[0144] Specifically, if Figure 5 As shown in the figure, the strategies for edge acquisition systems to report sensor data include:

[0145] 1) The edge acquisition system receives the sensor data reported by each sensor for cache, and receives its own working data. It packages its own working data in real time according to the protocol and sends it to the mobile terminal. For sensor data:

[0146] 2) Determine whether the cached sensor data meets the sending conditions, that is, whether it meets the reporting capacity requirements. If so, package the sensor data according to the protocol and send it to the mobile terminal;

[0147] 3) It is also determined whether the current time reaches the reporting period. If the reporting period is reached, the buffered (if any) sensor data is packaged and sent to the mobile terminal according to the protocol.

[0148] Example 6

[0149] This embodiment introduces a communication method, which includes:

[0150] Steps to collect data;

[0151] A step of compressing the collected data using the data compression method of embodiment 1 or the data compression unit of embodiment 2;

[0152] Steps for sending compressed data;

[0153] The step of receiving compressed data;

[0154] The step of decompressing the compressed data using the data decompression method of Example 3 or the data decompression unit of Example 4.

[0155] The above-mentioned process of sending and receiving compressed data uses satellite as the transmission medium. When the collected data meets the configured upload cycle or upload capacity threshold, the collected data is compressed. The mechanism can refer to the dual strategy control rule of the edge acquisition system reporting sensor data to the mobile terminal in Example 5.

[0156] Example 7

[0157] This embodiment uses 9588 pieces of sensor data for data compression testing. The total length of a single piece of data is 80 bytes. The compression rate of a single short string of data is compared. The results are as follows: Figure 6 shown.

[0158] Depend on Figure 6 It can be seen that due to the short character string, the effects of various compression algorithms are very unstable, and many negative compression rates occur, which is very unsatisfactory. Relatively speaking, the compression effect is greatly affected by the data content. The compression effect of the algorithm of the present invention on the test data is relatively better, and the positive compression rate is relatively more.

[0159] 9588 sensor data are combined, and every 10 data are integrated into a data packet, a total of 958 data packets, the total length of the single data is 656 bytes, the compression ratio is compared as follows: Figure 7 As shown. Figure 7 It can be seen that the compression rate of the algorithm of the present invention is completely stable at about 30% when compressing the integrated data, and even reaches 50% and 90% in some areas, which is a very ideal effect.

[0160] In addition, this embodiment also tests the single-thread compression performance.

[0161] 1. This compression algorithm

[0162] The compression algorithm was tested by cyclically compressing a single piece of data. The compression speed of the algorithm of the present invention can reach about 500,000 pieces / second, and the decompression speed can reach about 1.5 million pieces / second. CPU consumption: 16%

[0163] 2. zlib algorithm

[0164] The zlib compression algorithm was tested by looping and compressing a single data item. The compression speed can reach about 200,000 items / second, and the decompression speed can reach about 1.3 million items / second. CPU consumption: 19%.

[0165] Table 1. Comparison of compression / decompression rate and CPU consumption

[0166] algorithm This algorithm zlib Compression rate (10,000 pieces / second) 50 20 Decompression rate (10,000 pieces / second) 150 130 CPU consumption (%) 16 19

[0167] From the performance test point of view, the algorithm of the present invention has higher compression / decompression rate and less CPU consumption.

[0168] In terms of memory consumption:

[0169] 1. This compression algorithm

[0170] After expanding the 80-byte data packet content by 50,000 times, the test results shown in Table 2 were obtained.

[0171] Table 2 Thread memory consumption of this algorithm

[0172]

[0173]

[0174] It can be seen that after the data packet is expanded 50,000 times, the memory usage during compression is 0.2MB, and the memory usage during decompression is 3.8MB. The decompression process has increased significantly, but the memory usage is negligible. Figure 8 shown.

[0175] 2. zlib compression

[0176] After expanding the 80-byte data packet content by 50,000 times, the test results shown in Table 3 were obtained.

[0177] Table 3 zlib compression thread memory consumption

[0178] Line Mem usage Increment Occurrences 1 70.7MiB 70.7MiB 2 70.9MiB 0.2MiB compression 3 74.7MiB 4.5MiB Unzip 4 74.7MiB 0MiB

[0179] It can be seen that after the data packet was expanded 50,000 times, the memory usage during compression was 0.2MB, and the memory usage during decompression was 4.5MB.

[0180] In comparison, the data compression method of the present invention is closer to zlib compression in compression ratio, has lower resource consumption (CPU consumption), but has greatly improved compression / decompression rate, which has great advantages when running on low-performance terminals.

[0181] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A data compression method, wherein the data comprises at least one data block, characterized in that: At least one of the data blocks is compressed using the following method: Performing a first encoding on the data block to obtain first data and second data both including the maximum number of occurrences of the indicating character; When the maximum number of occurrences of a character in the first data reaches a second threshold, performing a first compression of the first data according to an indication of the maximum number of occurrences of a character in the first data; When the maximum number of occurrences of a character in the second data reaches a second threshold, performing a second compression on the second data according to an indication of the maximum number of occurrences of a character in the second data; A compression result of the data block is obtained.

2. The data compression method according to claim 1, characterized in that: Before performing the first encoding on the data block, whether to perform the first encoding on the data block is determined according to whether the length of the data block reaches a first threshold.

3. The data compression method according to claim 1, characterized in that: Performing a first compression on the first data includes: Pre-compressing the intermediate result of the first compression by a third compression; The compression performance of the pre-compression is calculated, and when the compression performance of the pre-compression satisfies the first condition, the intermediate result of the first compression is subjected to a third compression; the intermediate result indicates a mapping relationship between characters before and after the first compression.

4. The data compression method according to any one of claims 1 to 3, characterized in that: Also includes: Determine whether the length of the data after compression is reduced. If so, add an identifier indicating the compression process to the compressed data. If not, add an uncompressed identifier to the data.

5. The data compression method according to any one of claims 1 to 3, characterized in that: Also includes: Before compressing the data, it is determined whether the data satisfies the data type required for compression, and if so, the data is compressed.

6. A data decompression method for decompressing data compressed by the data compression method according to claim 4, characterized in that: include: Obtaining an identifier of the data, and determining whether the data is compressed according to the identifier; If the data is compressed, then according to the indication of the identifier, the compression result of each compressed data block of the data is decompressed using the following methods: According to the indication of the identifier, decompress the second data compressed in the compression result of the data block, or directly obtain the compressed second data; According to the indication of the identifier, decompressing the compressed first data in the compression result of the data block, or directly obtaining the compressed first data; The data obtained in the first two steps are subjected to a first decoding corresponding to the first encoding to obtain a decompression of the compression result of the data block.

7. A data compression unit, wherein the data comprises at least one data block, characterized in that: The data compression unit comprises: The encoding module is configured to: perform a first encoding on at least one of the data blocks to obtain first data and second data both including the maximum number of occurrences of the indicating character; A first compression module is configured to: when the maximum number of occurrences of a character in the first data reaches a second threshold, perform a first compression on the first data according to an indication of the maximum number of occurrences of a character in the first data; A second compression module is configured to: when the maximum number of times a character appears in the second data reaches a second threshold, perform a second compression on the second data according to an indication of the maximum number of times a character appears in the second data; The output module is configured to: obtain the output results of the first compression module and the second compression module, and output the compression result of the data block.

8. A data decompression unit, used for decompressing data compressed by the data compression method according to claim 4, characterized in that: The data decompression unit comprises: The identifier acquisition module is configured to: obtain an identifier of the data, and determine whether the data is compressed according to the identifier; if the data is determined to be compressed, enable the first decompression module, the second decompression module and the decoding module to decompress the compression result of each compressed data block of the data; A first decoding module is configured to: perform first compression decompression of the compressed first data in the compression result of the data block according to the instruction of the identifier, or directly obtain the compressed first data; The second decoding module is configured to: perform decompression of the second compressed data in the compression result of the data block according to the instruction of the identifier, or directly obtain the compressed second data; The decoding module is configured to: perform a first decoding corresponding to the first encoding on the data obtained by the first decoding module and the second decoding module to obtain a decompression result of the data block.

9. A mobile terminal, characterized in that: A data compression unit as claimed in claim 7 is provided.

10. The mobile terminal according to claim 9, characterized in that: It also includes a receiving unit for receiving sensor data, a preprocessing unit for preprocessing the sensor data, a format conversion unit for converting the sensor data into a data type that meets the compression requirements, and a sending unit for sending compressed data.

11. A cloud computing center, characterized in that: A data decompression unit as claimed in claim 8 is provided.

12. A communication system, characterized in that: It includes at least one mobile terminal as described in claim 9 or 10, and the cloud computing center as described in claim 11, each of the mobile terminals is respectively connected to the cloud computing center for communication, each of the mobile terminals compresses the acquired data using a data compression unit and sends the compressed data to the cloud computing center, and the cloud computing center decompresses the received data using a data decompression unit.

13. The communication system according to claim 12, characterized in that: The mobile terminal and the cloud computing center are connected in communication via a satellite.

14. The communication system according to claim 12 or 13, characterized in that: It also includes an edge acquisition system, the mobile terminal is connected to the edge acquisition system, the edge acquisition system is connected to multiple sensors, and the edge acquisition system aggregates the sensor data collected by each sensor and transmits it to the mobile terminal.

15. The communication system according to claim 14, characterized in that: The edge acquisition system transmits the aggregated sensor data to the mobile terminal when the configured upload cycle or upload capacity threshold is met.

16. A communication method, characterized in that: include: Collect data; Compressing the collected data using the data compression method described in any one of claims 1 to 5; Send compressed data; Receive compressed data; The compressed data is decompressed using the data decompression method described in claim 6.

17. The communication method according to claim 16, characterized in that: Satellite is used as a medium for sending and receiving compressed data.

18. The communication method according to claim 16 or 17, characterized in that: When the collected data meets the configured upload period or upload capacity threshold, the collected data is compressed.

Images