Asynchronous LZ77 optimization compression algorithm
Through asynchronous LZ77 optimization compression algorithm, the data structure and execution method are improved, and the problem of IO performance degradation in high-performance scenarios of DEFLATE algorithm is solved, achieving more efficient data processing and compatibility, and is suitable for complex application scenarios.
Patent Information
- Application Number
- CN202411115190.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing DEFLATE algorithms perform poorly in scenarios where large data streams or real-time read and write operations are required, resulting in degraded IO performance and increased resource usage, which cannot meet the needs of scenarios such as high-performance log writing and AI intelligent databases.
The asynchronous LZ77 optimization compression algorithm is adopted, and by improving the ternary data structure to (relative position, length, number of repetitions), using dictionary index to replace streaming position, and using asynchronous execution and B+ tree high-performance construction during data compression processing, decompression Tasks are processed in parallel, reducing IO operation blockage, monitoring dictionary resources and automatically cleaning.
It improves data readability and compatibility, supports asynchronous computing capabilities, reduces IO resource bottlenecks, improves IO performance and compression efficiency, and is suitable for scenarios such as high-performance log writing and AI intelligent databases.
Smart Images
Figure CN119109466A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer data processing and compression, in particular to an asynchronous LZ77 optimization compression algorithm. Background Art
[0002] Hardware technology in the IT field is constantly improving. Compression technology (compression algorithm) is an effective way to reduce IO and improve performance. It is widely used in compression software, data processing, embedded devices, etc. The currently popular HTTP / 2 (Hypertext Transfer Protocol version 2) and various transmission protocols also rely on the support of compression technology, which is an important part of optimizing the performance of transmission protocols.
[0003] In the existing compression technology industry, the mainstream technology has remained at the DEFLATE algorithm for many years. Popular rar, zip, gzip, and 7zip are all extensions based on the DEFLATE algorithm; the DEFLATE algorithm is a lossless data compression algorithm that combines the LZ77 algorithm and Huffman Coding. The design of this algorithm enables it to effectively compress data while maintaining data integrity.
[0004] The working principle of the DEFLATE algorithm mainly includes the following steps:
[0005] 1. LZ77 compression: First, the DEFLATE algorithm uses a variant of the LZ77 algorithm to find repeated strings in the input data. It uses a fixed-size sliding window (usually 32KB) to find the most recent repeated pattern.
[0006] 2. Dictionary encoding: The LZ77 algorithm generates a series of "dictionary" matches that represent repeated patterns in the data. Each match consists of a pointer (pointing to the previous occurrence of the string) and a length (the number of bytes matched).
[0007] 3. Huffman coding or fixed coding: The dictionary matches generated by LZ77 are then encoded as a series of bits. The DEFLATE algorithm can choose to use Huffman coding (a variable length coding scheme) or fixed coding (a fixed length coding scheme) to further compress the data.
[0008] 4. Storage and streaming: The DEFLATE algorithm is designed to handle data streams of arbitrary length, which makes it possible to use it for real-time compression and decompression.
[0009] The DEFLATE algorithm has the following shortcomings: Although the compression rate is improved by first compressing with the LZ77 algorithm and then with Huffman coding, this compression method limits its ability to achieve read-as-you-write (or instant read-write). Specifically:
[0010] Limitations of the compression process: The DEFLATE algorithm first uses the LZ77 algorithm to perform preliminary compression on the data. This step mainly reduces the data size by finding repeated data sequences (i.e. repeated strings within the "sliding window") and replacing them with shorter references (pointing to the location and length of the repeated data). Subsequently, the output of the LZ77 algorithm is further compressed using Huffman coding, which assigns different length codes according to the frequency of occurrence in the data to achieve more efficient compression.
[0011] Challenges of read-while-write: Since the DEFLATE algorithm involves two stages of compression (LZ77 and Huffman coding), and these two stages are usually performed sequentially, the entire compression process needs to wait until all data is fully processed before generating the final compressed data. This sequentiality and dependency makes the DEFLATE algorithm perform poorly in scenarios where large data streams or instant read and write operations are required. In particular, in scenarios where compression is required while writing, the DEFLATE algorithm may not be able to output the compressed data in time, thus failing to meet the needs of read-while-write.
[0012] At present, the application scenarios of technology are constantly increasing, and a single compression algorithm (such as DEFLATE) may not be able to meet all needs, nor support the ever-changing industry needs. For example:
[0013] High-performance log writing, AI intelligent database. The log writing requirements of business servers are increasing, and the log writing performance bottleneck is gradually emerging.
[0014] As containerization technology continues to mature, the computing unit model is also becoming more and more perfect. The application of sidecar mode, queues and other solutions has significantly enhanced log writing and computing processing capabilities. However, IO resources often become the performance bottleneck of the server, limiting the further improvement of overall performance. For scenarios that require high-performance log writing or similar applications, IO resources often become a performance bottleneck. This shortcoming of the DEFLATE algorithm may cause data to be delayed during the writing process due to waiting for compression, thereby reducing the overall IO performance. It is mainly reflected in the following aspects:
[0015] Compression delay: DEFLATE is a relatively complex compression algorithm, which involves two steps: LZ77 algorithm and Huffman coding. When processing large-scale data, the execution of these two steps will consume a certain amount of time, resulting in the need to wait for compression to complete during the data writing process. This delay will increase the response time of IO operations, thereby reducing the overall IO performance.
[0016] Resource usage: The DEFLATE algorithm will occupy a certain amount of CPU and memory resources during the compression process. For IO-intensive applications, the usage of these resources may further aggravate the resource shortage of the system and affect the execution efficiency and performance of other IO operations.
[0017] Balance between compression efficiency and IO performance: Although the DEFLATE algorithm can provide a high compression rate, it may also sacrifice IO performance while pursuing compression efficiency. In some scenarios with extremely high IO performance requirements, the improvement in compression rate may not be enough to compensate for the loss caused by the decline in IO performance.
[0018] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0019] In view of the defects existing in the prior art, the purpose of the present invention is to provide an asynchronous LZ77 optimized compression algorithm, which optimizes the data structure by improving the ternary data structure, enhances data readability and compatibility, supports asynchronous computing capabilities, and not only improves IO performance, but also improves compression efficiency, and is suitable for more application scenarios.
[0020] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0021] An asynchronous Lz77 optimized compression algorithm, characterized by comprising:
[0022] Optimize the data structure, use (relative position, length, number of repetitions) as the ternary data structure, and use dictionary index instead of streaming position;
[0023] The structure of the dictionary index is (Index, CurrentChar), where index is the dictionary index and CurrentChar is the current character;
[0024] The data compression process is performed asynchronously and uses plain text.
[0025] On the basis of the above technical solution, after optimizing the data structure, the following full-text replacement operation is further performed: replace "(" with "\(.", replace "\" with "\\.".
[0026] On the basis of the above technical solution, during data compression processing, the B+ tree high performance is called to build the compression format, and the memory dictionary structure of the b+ tree is asynchronously built.
[0027] On the basis of the above technical solution, the asynchronous execution specifically includes: during the compression process, decomposing the compression task into multiple small tasks, and using an asynchronous task scheduler to manage these tasks.
[0028] On the basis of the above technical solution, when reading the data to be compressed, asynchronous IO operation is used to read the file or network data stream non-blockingly; when the data is available, the task of processing the data is automatically triggered;
[0029] After compression is completed, asynchronous IO operations are also used to write data to the target file or network transmission stream; this can reduce the time the main thread is blocked due to IO operations.
[0030] Based on the above technical solution, multiple data blocks are processed in parallel using multithreading or coroutines to improve compression efficiency;
[0031] After all data blocks are compressed, the results are merged into the final compressed file.
[0032] Based on the above technical solution, during the compression process, when a new character sequence is found to be added to the dictionary, a B+ tree index is asynchronously constructed or updated to support efficient search and matching operations;
[0033] Asynchronously monitor the usage of dictionary resources. When the upper limit is approached, the dictionary cleanup or optimization operation is automatically triggered to maintain efficient operation.
[0034] On the basis of the above technical solution, an error handling mechanism is set for each asynchronous task, and when a task fails, the error can be automatically recorded, retried or rolled back;
[0035] State synchronization is used to ensure state synchronization between all asynchronous tasks to avoid data inconsistency or loss.
[0036] Based on the above technical solution, the node structure of the B+ tree is defined as follows:
[0037] Node type, leaf node or non-leaf node;
[0038] Key, that is, dictionary index key;
[0039] Pointer, child node pointer or data pointer;
[0040] Asynchronously initialize the B+ tree dictionary, including allocating memory space and setting the root node;
[0041] When the compression algorithm processes the data stream and identifies a new compressible pattern, an asynchronous insert operation is triggered, which attempts to add the new pattern to the B+ tree dictionary;
[0042] Locks are used to ensure data consistency and integrity during the insertion process.
[0043] Based on the above technical solution, the size of the dictionary is monitored asynchronously, and a cleanup operation is triggered when necessary, with the least used entries being deleted first.
[0044] The asynchronous LZ77 optimization compression algorithm described in the present invention has the following beneficial effects:
[0045] 1. Optimize data structure, including:
[0046] Improved triple data structure: from the traditional (relative position, length, current character) triple structure, optimized to (relative position, length, number of repetitions), increased the ability to handle repeated content;
[0047] Dictionary index replaces streaming position: Drawing on the LZ78 algorithm, dictionary index is used instead of streaming position to improve the efficiency of data retrieval and compression.
[0048] 2. Enhance data readability and compatibility, including:
[0049] Plain text compression: The optimized algorithm is plain text, which means that the compressed data still maintains the readability of the original text and is compatible with mainstream text editing tools;
[0050] Improved compatibility: This feature allows the algorithm to be seamlessly integrated in a variety of application scenarios without having to worry about data format compatibility issues.
[0051] 3. Support asynchronous computing capabilities, including:
[0052] Asynchronous processing capability: The traditional lz77 algorithm is a blocking synchronous algorithm, while the optimized algorithm has asynchronous computing capability and can process data more efficiently;
[0053] Reduce latency: Asynchronous computing can reduce waiting time during processing and improve the response speed and efficiency of the overall system.
[0054] 4. Improve IO performance, including:
[0055] Reduce IO resource bottlenecks: With the increasing demand for server log writing, traditional compression algorithms encounter bottlenecks in IO performance. The optimized asynchronous Lz77 algorithm effectively improves IO writing performance by reducing the damage to data throughput capacity.
[0056] Support high-performance scenarios: It is particularly suitable for scenarios such as high-performance log writing and AI intelligent databases, meeting the growing business needs.
[0057] 5. Improve compression efficiency, including:
[0058] Repetition number expression: By introducing the repetition number expression method, repeated content can be better processed, thereby improving compression efficiency;
[0059] Efficient B+ tree construction: The memory dictionary structure of the B+ tree is constructed asynchronously to further improve the construction speed and performance of the compressed format.
[0060] The optimized algorithm is not only applicable to traditional compression scenarios, but also can meet increasingly complex business needs, such as solutions such as sidecars and queues in containerization technology. The algorithm is designed to be scalable and can be iterated and optimized as technology develops and application scenarios change. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] The present invention has the following accompanying drawings:
[0062] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0063] Figure 1 A flowchart of Embodiment 1 of an asynchronous LZ77 optimization compression algorithm described in the present invention. DETAILED DESCRIPTION
[0064] The present invention is further described in detail below in conjunction with the accompanying drawings. The detailed description is an explanation made in conjunction with the exemplary embodiments of the present invention, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0065] like Figure 1 As shown, the present invention provides an asynchronous LZ77 optimization compression algorithm, including:
[0066] Optimize the data structure, use (relative position, length, number of repetitions) as the ternary data structure, and use dictionary index instead of streaming position;
[0067] The structure of the dictionary index is (Index, CurrentChar), where index is the dictionary index and CurrentChar is the current character;
[0068] Furthermore, after optimizing the data structure, the following full-text replacement operations are performed: replace "(" with "\(.", replace "\" with "\\."; the purpose of the replacement is to avoid syntax conflicts and avoid triggering any formatting commands or special behaviors;
[0069] In the original LZ77 algorithm, the ternary data structure is (relative position, length, current character); this embodiment optimizes it to (relative position, length, number of repetitions), introduces the concept of number of repetitions to better handle repeated content and improve compression efficiency. When repeated content appears, it can be repeated all the time;
[0070] The purpose of using dictionary index instead of streaming position is to improve the efficiency of data retrieval and compression. This part can refer to the LZ78 algorithm;
[0071] The data compression process is performed asynchronously and in plain text;
[0072] The use of asynchronous execution improves computing power, causes very little damage to data throughput, and improves IO write performance; the compression algorithm uses plain text, which improves compatibility and is compatible with all mainstream text editing tools.
[0073] On the basis of the above technical solution, during data compression processing, the B+ tree high performance is called to build the compression format, and the memory dictionary structure of the b+ tree is asynchronously built.
[0074] The performance of building the compression format is improved through the memory dictionary of the B+ tree structure. Using the high-performance B+ tree to build the compression format and asynchronously building its memory dictionary structure can significantly improve the efficiency of data indexing and retrieval, optimize storage space, and improve the compression ratio. At the same time, the asynchronous processing method reduces the burden on the main thread and enhances the responsiveness of the system. In addition, the stability and scalability of the B+ tree also provide a strong guarantee for the application of this technical solution in the field of large-scale data processing.
[0075] Based on the above technical solution, the node structure of the B+ tree is defined as follows:
[0076] Node type, leaf node or non-leaf node;
[0077] Key, that is, dictionary index key;
[0078] Pointer, child node pointer or data pointer;
[0079] Asynchronously initialize the B+ tree dictionary, including allocating memory space and setting the root node;
[0080] When the compression algorithm processes the data stream and identifies a new compressible pattern (i.e., a new sequence of characters), an asynchronous insert operation is triggered, which attempts to add the new pattern to the B+ tree dictionary;
[0081] Use locks (such as read-write locks) to ensure data consistency and integrity during the insertion process.
[0082] Based on the above technical solution, the size of the dictionary is monitored asynchronously, and a cleanup operation is triggered when necessary, with the least used entries being deleted first.
[0083] This embodiment is an effective strategy for optimizing system resource utilization and performance. By reducing memory usage, optimizing response time, preventing memory leaks, and improving cache hit rate, this method can significantly improve system stability and response speed, while simplifying code logic for easy expansion and maintenance. It is applicable to a variety of scenarios, including cache systems, log processing, and state management, and helps improve user experience and overall application performance.
[0084] On the basis of the above technical solution, the asynchronous execution specifically includes: during the compression process, decomposing the compression task into multiple small tasks, and using an asynchronous task scheduler to manage these tasks.
[0085] On the basis of the above technical solution, when reading the data to be compressed, asynchronous IO operation is used to read the file or network data stream non-blockingly; when the data is available, the task of processing the data is automatically triggered;
[0086] After compression is completed, asynchronous IO operations are also used to write data to the target file or network transmission stream; this can reduce the time the main thread is blocked due to IO operations.
[0087] Based on the above technical solution, multiple data blocks are processed in parallel using multithreading or coroutines to improve compression efficiency;
[0088] After all data blocks are compressed, the results are merged into the final compressed file.
[0089] Based on the above technical solution, during the compression process, when a new character sequence is found to be added to the dictionary, a B+ tree index is asynchronously constructed or updated to support efficient search and matching operations;
[0090] Asynchronously monitor the usage of dictionary resources. When the upper limit is approached, the dictionary cleanup or optimization operation is automatically triggered to maintain efficient operation.
[0091] On the basis of the above technical solution, an error handling mechanism is set for each asynchronous task, and when a task fails, the error can be automatically recorded, retried or rolled back;
[0092] State synchronization is used to ensure state synchronization between all asynchronous tasks to avoid data inconsistency or loss.
[0093] The following is a specific example.
[0094] Taking the input text "abcdefgabcdefgabcdefgabcdefgabcdefg" as an example, the compressed text is "abcdefg(6,6,5)", where 6 represents the relative position, 6 represents the length, and 5 represents the number of repetitions.
[0095] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
[0096] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. An asynchronous Lz77 optimized compression algorithm, characterized in that: include: The data structure is optimized, using (relative position, length, number of repetitions) as the ternary data structure, and using dictionary indexes instead of streaming positions; The structure of the dictionary index is (Index, CurrentChar), where index is the dictionary index and CurrentChar is the current character; The data compression process is performed asynchronously and uses plain text.
2. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: After optimizing the data structure, further perform the following full-text replacement operations: replace "(" with "\(.", replace "\" with "\\.".
3. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: When data is compressed, the B+ tree is called to build the compression format with high performance, and the memory dictionary structure of the b+ tree is built asynchronously.
4. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: The asynchronous execution specifically includes: during the compression process, decomposing the compression task into multiple small tasks, and using an asynchronous task scheduler to manage these tasks.
5. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: When reading the data to be compressed, use asynchronous IO operations to read files or network data streams non-blockingly; when data is available, automatically trigger the task of processing the data; After compression is completed, asynchronous IO operations are also used to write data to the target file or network transmission stream; this can reduce the time the main thread is blocked due to IO operations.
6. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: Use multithreading or coroutines to process multiple data blocks in parallel to improve compression efficiency; After all data blocks are compressed, the results are merged into the final compressed file.
7. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: During the compression process, when a new character sequence is found that needs to be added to the dictionary, the B+ tree index is asynchronously built or updated to support efficient search and matching operations; Asynchronously monitor the usage of dictionary resources. When the upper limit is approached, the dictionary cleanup or optimization operation is automatically triggered to maintain efficient operation.
8. An asynchronous Lz77 optimized compression algorithm as claimed in claim 1, characterized in that: Set up an error handling mechanism for each asynchronous task, so that when a task fails, it can automatically record the error, retry or roll back. State synchronization is used to ensure state synchronization between all asynchronous tasks to avoid data inconsistency or loss.
9. An asynchronous Lz77 optimized compression algorithm as claimed in claim 3, characterized in that: The node structure of the B+ tree is defined as follows: Node type, leaf node or non-leaf node; Key, that is, dictionary index key; Pointer, child node pointer or data pointer; Asynchronously initialize the B+ tree dictionary, including allocating memory space and setting the root node; When the compression algorithm processes the data stream and identifies a new compressible pattern, an asynchronous insert operation is triggered, which attempts to add the new pattern to the B+ tree dictionary; Locks are used to ensure data consistency and integrity during the insertion process.
10. An asynchronous Lz77 optimized compression algorithm as claimed in claim 3, characterized in that: Asynchronously monitors the size of the dictionary and triggers a cleanup operation when necessary, preferentially removing the least used entries.