Sparse data compression method based on bitmap dictionary and related equipment

Abstract

The invention discloses a sparse data compression method based on a bitmap dictionary and related equipment, and belongs to the technical field of data compression, and the method comprises the following steps: defining a default value according to an original data sequence; traversing the original data sequence, marking data units in the original data sequence as a first state or a second state according to a default value to generate a bitmap dictionary, and defining one of the first state and the second state as an effective state; extracting data units marked as valid states in the original data sequence, and forming valid data loads according to the original sequence; and packaging the bitmap dictionary and the effective data load according to a predetermined format to form a compressed data block. According to the method, distribution of effective data is identified by introducing the bitmap dictionary, sparse data compression and decompression with extremely high compression ratio and extremely high encoding and decoding speed are realized, and the storage and transmission overhead of a large number of repeated default values is eliminated.

Description

Technical Field

[0001] This application relates to the field of data compression technology, specifically to a sparse data compression method and related equipment based on a bitmap dictionary. Background Technology

[0002] In the field of information technology, data compression is a key technology for saving storage space and improving transmission efficiency. Existing technologies include various general-purpose compression algorithms, such as LZ77, LZ78, and Huffman coding. However, these algorithms do not achieve optimal efficiency when processing a special type of data—sparse data.

[0003] Sparse data refers to data sequences containing a large number of identical values ​​(such as 0, NULL, or specific default values), with only a few positions containing valid non-default values. This type of data is commonly found in: scientific computing: matrices that remain unchanged in most areas during large-scale numerical simulations; IoT sensor data: repetitive values ​​continuously uploaded by sensors under steady-state conditions; image processing: sparse or binary images; and database storage: sparse columns where most records are empty.

[0004] Existing general-purpose compression algorithms still require computation and encoding of each data unit when compressing sparse data, even if the data consists of many repetitive default values. This leads to unnecessary computational overhead, and there is still room for improvement in compression ratio. For example, for a sequence containing 8 bytes, most of which are 0x00, general-purpose algorithms still need to process all 8 bytes, failing to implement a "skip" mechanism at the data structure level. Therefore, there is an urgent need in the field for a dedicated compression method that can directly identify and utilize the sparsity characteristics of data to achieve higher compression ratios and faster processing speeds. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a sparse data compression method and related equipment based on a bitmap dictionary. By introducing a bitmap dictionary to identify the distribution of effective data, the storage and transmission overhead of a large number of repetitive default values ​​is eliminated.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides a sparse data compression method based on a bitmap dictionary, comprising the following steps: Receive the raw data sequence to be compressed, the raw data sequence comprising a number of data units; Define default values ​​based on the original data sequence; Traverse the original data sequence and mark the data units in the original data sequence as a first state or a second state according to the default value to generate a bitmap dictionary. One of the first state and the second state is defined as a valid state. Extract the data units marked as valid states from the original data sequence and assemble them into a valid data payload in their original order; The bitmap dictionary and the effective data payload are encapsulated in a predetermined format to form a compressed data block.

[0007] Furthermore, the method also includes decompressing the compressed data block, specifically: Receive and parse the compressed data block, separating the bitmap dictionary and the valid data payload; Construct an output sequence and initialize all positions in the output sequence to the default values; The bitmap dictionary is read sequentially. Whenever a valid bit is read, a data unit is taken out from the valid data payload in sequence and filled into the corresponding position in the output sequence. Once the bitmap dictionary has been traversed, the decompressed data sequence is generated and output.

[0008] Furthermore, the output sequence, bitmap dictionary, and original data sequence have the same length.

[0009] Further, the step of traversing the original data sequence and marking the data units in the original data sequence as a first state or a second state according to the default value is used to generate a bitmap dictionary. The traversal rule is as follows: in the original data sequence, if the value of a data unit is equal to the default value, the corresponding data unit is marked as the first state; otherwise, the corresponding data unit is marked as the second state. The data units are marked in sequence.

[0010] Furthermore, the step of encapsulating the bitmap dictionary and the effective data payload in a predetermined format to form a compressed data block specifically involves encapsulating the bitmap dictionary and the effective data payload in sequence.

[0011] Further, the original data sequence is traversed, and data units in the original data sequence are marked as a first state or a second state according to the default value to generate a bitmap dictionary, including: Traverse the original data sequence and, according to the default value, mark the data units in the original data sequence as a first state or a second state to generate a bitmap dictionary in binary form; The bitmap dictionary in binary form is converted into bytes and stored; During the decompression process, the bitmap dictionary in byte form is converted into binary form before sequentially reading the bitmap dictionary.

[0012] Furthermore, defining a default value based on the original data sequence specifically involves: Identify a number from the original data sequence and define it as the default value.

[0013] Furthermore, defining a default value based on the original data sequence specifically involves defining the number that appears most frequently in the original data sequence as the default value.

[0014] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the sparse data compression method based on a bitmap dictionary as described in any of the preceding claims.

[0015] Thirdly, the present invention provides a computer device, comprising: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of any of the preceding bitmap dictionary-based sparse data compression methods.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This invention provides a sparse data compression method based on a bitmap dictionary. It defines a default value based on the original data sequence, generates a bitmap dictionary based on the default value, and then obtains the effective data payload from the original data sequence according to the bitmap dictionary. The bitmap dictionary and the effective data payload are then encapsulated in a predetermined format to form a compressed data block. This invention introduces a bitmap dictionary to identify the distribution of effective data, providing a supplement to existing compression algorithms for sparse data compression and transmission. It achieves extremely high compression ratios and extremely fast encoding and decoding speeds for sparse data compression and decompression, eliminating the storage and transmission overhead of a large number of repetitive default values. Attached Figure Description

[0017] Figure 1 This is a flowchart of a sparse data compression method based on a bitmap dictionary provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances. Example

[0021] Please see Figure 1 This embodiment introduces a sparse data compression and decompression method based on a bitmap dictionary, including the following steps: S1. Compression process: Data sequence input: Receives the raw data sequence to be compressed, which consists of N data units.

[0022] Default value definition: A default value (such as 0x00) is preset or identified from the data. In other embodiments, the most frequently occurring number may also be defined as the default value.

[0023] Bitmap dictionary generation: Traverse the original data sequence to generate a bitmap dictionary of length N.

[0024] Specifically, the traversal rule is as follows: if the value of the i-th data unit is equal to the default value, then the i-th bit in the bitmap dictionary is set to the first state (e.g., '0'); if it is not equal to the default value, then it is set to the second state (e.g., '1').

[0025] Valid data extraction: Extract all data units marked as the second state ('1') in the bitmap dictionary from the original data sequence and assemble them into a valid data payload in their original order. In this embodiment, the second state is the valid state.

[0026] Data encapsulation: The bitmap dictionary and the effective data payload are encapsulated in a predetermined format to form a compressed data block.

[0027] S2. Decompression process: Data parsing: Parse compressed data blocks to separate the bitmap dictionary and the effective data payload.

[0028] Output sequence initialization: Create an output sequence of length N (determined by the length of the bitmap dictionary) and initialize all positions to their default values.

[0029] Data restoration: Read each bit of the bitmap dictionary sequentially. When a bit in the second state ('1') is read, take a data unit from the valid data payload in sequence and fill it into the corresponding position in the output sequence.

[0030] Sequence Output: Once the bitmap dictionary has been traversed, a complete, restored data sequence is generated and output.

[0031] The following describes the content involved in the above embodiments with reference to a preferred embodiment, including the following steps: S100, Compression process: S110, Data Sequence Input: Input the original data sequence, which consists of 8 data units.

[0032] S120, Bitmap dictionary generation: S121, Traverse sequence: 0x00 (default)->0, 0x00 (default)->0, 0x12 (valid)->1, 0x32 (valid)->1, the next 4 0x00 (default)->0000.

[0033] S122. The generated bitmap dictionary (binary) is 00110000.

[0034] S123. For ease of storage, convert it to one byte: 0x30.

[0035] S130. Valid data extraction: Extract the corresponding data [0x12, 0x32] based on the position of '1' in the bitmap dictionary (the 3rd and 4th bits).

[0036] S140, Encapsulation: Encapsulate the bitmap dictionary 0x30 and the valid data payload 0x12, 0x32 in sequence.

[0037] The final compressed data is: [0x30, 0x12, 0x32].

[0038] S200, Decompression process: S210, Data parsing: Read the compressed data, identify the bitmap dictionary as 0x30, and the effective data payload as [0x12, 0x32].

[0039] S220, Output sequence initialization: Create an output buffer of length 8 and initialize it to the default value 0x00. The buffer is now: [0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00].

[0040] S230, Data Restoration: S231. Convert the bitmap dictionary 0x30 to binary 00110000 (assuming that the sequence indexes from the high bit to the low bit are 1 to 8).

[0041] S232: Sequential scanning of bitmap dictionary: If the first and second bits are '0', skip them and do not modify the buffer.

[0042] The third bit is '1'. Take the first data 0x12 from the payload and fill it into the third bit of the buffer.

[0043] The 4th bit is '1'. Take the second data 0x32 from the payload and fill it into the 4th bit of the buffer.

[0044] If the 5th to 8th digits are '0', skip them.

[0045] S240, Sequence Output: The final restored data sequence is obtained and output as: [0x00,0x00,0x12,0x32,0x00,0x00,0x00,0x00].

[0046] The restored data sequence is completely identical to the original data sequence, achieving lossless decompression.

[0047] In summary, this invention introduces a bitmap dictionary to identify the distribution of valid data, which can directly identify and utilize the sparsity of data to achieve a higher compression ratio and faster processing speed. It provides a certain supplement to the compression and transmission of sparse data in existing compression algorithms, thereby achieving sparse data compression and decompression with extremely high compression ratio and extremely fast encoding and decoding speed, and eliminating the storage and transmission overhead of a large number of repetitive default values.

[0048] Example 2: This example provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in Example 1.

[0049] Example 3: This example provides a computer device, including: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of any of the methods described in Embodiment 1.

[0050] Example 4: This example provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method described in any one of Examples 1.

[0051] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0052] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit its protection scope. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading this disclosure, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims.

Claims

1. A sparse data compression method based on a bitmap dictionary, characterized in that, Includes the following steps: Receive the raw data sequence to be compressed, the raw data sequence comprising a number of data units; Define default values ​​based on the original data sequence; Traverse the original data sequence and mark the data units in the original data sequence as a first state or a second state according to the default value to generate a bitmap dictionary. One of the first state and the second state is defined as a valid state. Extract the data units marked as valid states from the original data sequence and assemble them into a valid data payload in their original order; The bitmap dictionary and the effective data payload are encapsulated in a predetermined format to form a compressed data block.

2. The sparse data compression method based on bitmap dictionary according to claim 1, characterized in that, The method further includes decompressing the compressed data block, specifically: Receive and parse the compressed data block, separating the bitmap dictionary and the valid data payload; Construct an output sequence and initialize all positions in the output sequence to the default values; The bitmap dictionary is read sequentially. Whenever a valid bit is read, a data unit is taken out from the valid data payload in sequence and filled into the corresponding position in the output sequence. Once the bitmap dictionary has been traversed, the decompressed data sequence is generated and output.

3. The sparse data compression method based on bitmap dictionary according to claim 2, characterized in that, The output sequence, bitmap dictionary, and original data sequence have the same length.

4. The sparse data compression method based on bitmap dictionary according to claim 1, characterized in that, The process involves traversing the original data sequence and marking data units in the original data sequence as either a first state or a second state according to the default value to generate a bitmap dictionary. The traversal rule is as follows: in the original data sequence, if the value of a data unit is equal to the default value, the corresponding data unit is marked as the first state; otherwise, the corresponding data unit is marked as the second state. The data units are marked sequentially.

5. The sparse data compression method based on bitmap dictionary according to claim 1, characterized in that, The step of encapsulating the bitmap dictionary and the effective data payload in a predetermined format to form a compressed data block specifically involves encapsulating the bitmap dictionary and the effective data payload in sequence.

6. The sparse data compression method based on bitmap dictionary according to claim 2, characterized in that, Traversing the original data sequence, and marking data units in the original data sequence as a first state or a second state according to the default value, to generate a bitmap dictionary, including: Traverse the original data sequence and, according to the default value, mark the data units in the original data sequence as a first state or a second state to generate a bitmap dictionary in binary form; The bitmap dictionary in binary form is converted into bytes and stored; During the decompression process, the bitmap dictionary in byte form is converted into binary form before sequentially reading the bitmap dictionary.

7. The sparse data compression method based on bitmap dictionary according to claim 1, characterized in that, The definition of a default value based on the original data sequence is as follows: Identify a number from the original data sequence and define it as the default value.

8. The sparse data compression method based on bitmap dictionary according to claim 1, characterized in that, The step of defining a default value based on the original data sequence specifically involves defining the number that appears most frequently in the original data sequence as the default value.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the sparse data compression method based on a bitmap dictionary as described in any one of claims 1 to 8.

10. A computer device, characterized in that, include: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of the bitmap dictionary-based sparse data compression method as described in any one of claims 1 to 8.

Images