A Distributed Storage Data Compression Method

Abstract

This invention relates to a distributed storage data compression method and apparatus. The method includes the steps of pre-compressing the original data blocks before data is written to disk to obtain pre-compressed data blocks, and then compacting and arranging the pre-compressed data blocks to obtain compacted data blocks. By dividing the compression process of the original data blocks into pre-compression and compaction / arrangement, and further compacting and arranging the data blocks based on pre-compression, this method significantly improves the data compression ratio while achieving a balance between performance loss and data compression ratio. This method addresses the problem that existing compression schemes in the distributed storage field cannot effectively balance performance loss and data compression ratio, and that the data compression ratio is generally low due to efforts to reduce data read / write latency. This method has a simple architecture, is easy to implement, and offers high security and reliability, contributing to the performance improvement of distributed storage systems.

Description

Technical Field

[0001] This invention belongs to the technical field of data compression methods, and particularly relates to a distributed storage data compression method and apparatus. Background Technology

[0002] A distributed storage system is a data storage system that distributes data across multiple independent devices. Traditional network storage systems use a centralized storage server to store all data, making the storage server a bottleneck for system performance. Because the reliability and security of storage servers are difficult to guarantee, they cannot meet the needs of large-scale storage applications. In contrast, distributed network storage systems employ a scalable system architecture, utilizing multiple storage servers to share the storage load and location servers to locate stored information. This not only improves the system's reliability, availability, and access efficiency but also facilitates expansion.

[0003] Data compression is divided into lossy compression and lossless compression. In image processing, lossy compression does not significantly affect the appearance of the image, so either lossy or lossless compression can be chosen based on the required image accuracy. However, in distributed storage, data loss is unacceptable, therefore distributed storage systems must employ lossless compression. Currently, the compression algorithms used in the compression solutions of mainstream storage vendors are largely similar, consisting of generally recognized effective algorithms such as Gzip and LZ4. The differences between compression solutions from different vendors mainly lie in the specific implementation of the compression scheme.

[0004] We know that a good compression scheme needs to balance performance loss and data compression rate. Currently, compression schemes in the distributed storage field mainly suffer from the following problems:

[0005] (1) Low data compression rate

[0006] For distributed storage systems, if data compression is enabled, each piece of data will be compressed before being written to disk. Furthermore, the higher the compression ratio of the algorithm used, the longer the compression process will take, ultimately affecting the efficiency of data writing to disk. Similarly, the time required to decompress data during reading will also be affected. Therefore, enabling high-compression-ratio data compression will significantly increase data read / write latency. Considering the performance requirements of customer businesses, it is necessary to reduce data read / write latency in the data processing flow. Actual compression schemes often use compression algorithms with relatively low compression ratios, resulting in insufficient data compression.

[0007] (2) Cannot balance performance loss and data compression rate

[0008] Compression solutions prioritize different aspects depending on the business scenario. For real-time scenarios, stable and low-latency performance is crucial, directly impacting customer experience and even service availability. Therefore, compression solutions often employ algorithms with relatively low compression ratios, achieving a certain level of compression while minimizing performance impact on the customer's business. Conversely, for backup scenarios, space reduction is more important than low latency, directly saving costs. Thus, compression solutions typically use algorithms with relatively high compression ratios, maximizing compression while ensuring backups can be completed within the backup window. However, current distributed storage compression solutions do not achieve a good dynamic balance between performance loss and data compression ratio, thus affecting the customer experience. Summary of the Invention

[0009] To address the problem that existing compression schemes in the distributed storage field fail to adequately balance performance loss and data compression rate, and that attempts to reduce data read / write latency often result in generally low data compression rates, we offer a solution. This invention's data compression scheme divides the overall data compression process into two stages: pre-compression and compaction arrangement. This achieves a balance between performance loss and data compression rate, while simultaneously improving the data compression rate.

[0010] The overall process of this solution is as follows: Before the data is pushed to disk, the original data blocks first enter the pre-compression process. The original data blocks are pre-compressed into pre-compressed data blocks. Subsequently, the pre-compressed data blocks are further compressed and arranged. Finally, multiple original data blocks are arranged together, thereby reducing the space occupied by the original data blocks.

[0011] Specifically, in a first aspect, the present invention provides a distributed storage data compression method, the method comprising:

[0012] S1. Before the data is written to disk, the original data blocks are pre-compressed to obtain pre-compressed data blocks;

[0013] S2. Calculate the pre-compression gain of the pre-compressed data block obtained in the previous step and compare it with a preset threshold. If the pre-compression gain is greater than or equal to the preset threshold, the pre-compressed data block will continue to enter the next step of the compaction and arrangement process.

[0014] S3. Compact and arrange the pre-compressed data blocks to obtain compressed data blocks.

[0015] Furthermore, according to some embodiments of the present invention, the pre-compression process in step S1 of the distributed storage data compression method of the present invention is as follows:

[0016] 1) If a hardware acceleration card is detected in the storage system, a hardware pre-compression process is performed;

[0017] 2) If no hardware acceleration card is detected in the storage system, proceed with the software pre-compression process;

[0018] 3) The software and hardware pre-compression results are compatible with each other.

[0019] Furthermore, according to some embodiments of the present invention, the pre-compressed data block in step S1 of the distributed storage data compression method of the present invention includes a compression header, a data segment, and a key corresponding to the original data block;

[0020] The compression header contains the compression algorithm used during the pre-compression of the original data blocks, the data segment length, and the key length information.

[0021] Furthermore, according to some embodiments of the present invention, the pre-compression gain in step S2 of the distributed storage data compression method of the present invention is the difference between the length of the original data block and the length of the corresponding pre-compressed data block. If the pre-compression gain is greater than or equal to a preset threshold, the pre-compressed data block continues to enter the compaction and arrangement process; if the pre-compression gain is less than the preset threshold, it means that the additional metadata brought by the pre-compression operation will occupy more space, so the current pre-compression process needs to be terminated, and the original data block is not compressed and is directly removed from the disk. Considering scalability, the threshold of the pre-compression gain can be dynamically configured.

[0022] Furthermore, according to some embodiments of the present invention, step S3 of the distributed storage data compression method of the present invention, which involves compressing and arranging pre-compressed data blocks to obtain compressed data blocks, includes:

[0023] S31. Pre-compressed data blocks entering the compaction and arrangement process are placed into cache queues of corresponding sizes according to their different sizes;

[0024] S32. After the pre-compressed data block enters the cache queue, the system starts a coroutine. This coroutine retrieves the pre-compressed data block from the cache queue and puts it into the processing queue. Then, it retrieves the pre-compressed data block from the processing queue and compresses and arranges it.

[0025] S33. When the amount of pre-compressed data blocks to be processed in the cache queue reaches a preset threshold, the system starts a new coroutine to quickly process the pre-compressed data blocks backlogged in the cache queue. At the same time, considering that too many coroutine tasks will impact system performance, the system needs to control the upper limit of the number of coroutines, which can be dynamically configured. When the amount of pre-compressed data blocks to be processed in the cache queue decreases to below the preset threshold, and this state continues for a period of time (the threshold for the duration can be dynamically configured), the system stops the corresponding number of coroutines to reduce the impact of too many coroutine tasks on system performance.

[0026] Furthermore, according to some embodiments of the present invention, the cache queues mentioned in step S31 of the above-mentioned distributed storage data compression method are 10 in total, which are used to cache pre-compressed data blocks with sizes less than or equal to 8K, 4K, 2K, 1K, 512B, 256B, 128B, 64B, 32B, and 16B respectively; for example, if the size of the pre-compressed data block is 6K, it will enter the 8K cache queue, and if the size of the pre-compressed data block is 800B, it will enter the 1K cache queue.

[0027] In step S33, when the amount of pre-compressed data blocks to be processed in the cache queue reaches a preset threshold, the system starts a new coroutine to quickly process the pre-compressed data blocks backed up in the cache queue. Since there may be multiple coroutines in the system at the same time, in order to avoid mutual exclusion and preemption of processing queues between coroutines and affect performance, each coroutine has its own exclusive processing queue. The processing queues correspond one-to-one with the cache queues, and there are 10 in total, which store pre-compressed data blocks from different cache queues.

[0028] Furthermore, according to some embodiments of the present invention, after the pre-compressed data block enters the cache queue in step S32 of the above-described distributed storage data compression method, the system starts a coroutine. This coroutine retrieves the pre-compressed data block from the cache queue and places it into the processing queue. Then, it retrieves the pre-compressed data block from the processing queue and compresses and arranges it. The steps for compressing and arranging the pre-compressed data block are as follows:

[0029] S321. The coroutine scans all processing queues in a loop, following the order from the 8K processing queue to the 16B processing queue.

[0030] S322. Based on the remaining space size of the compressed data block, the coroutine retrieves the next processing queue to be scanned that is less than or equal to the remaining space size, and then directly jumps to that processing queue to obtain the pre-compressed data block for compaction and arrangement. The direct jump method can skip processing queues that do not meet the conditions, reducing wasted work; for example, if the initial remaining space size of the compressed data block is 8K, the coroutine jumps to the 8K processing queue to obtain the pre-compressed data block. After all the pre-compressed data blocks in the 8K processing queue are compacted and arranged into the compressed data block, the remaining space size of the compressed data block is reduced to 1000B. At this time, the coroutine does not need to scan the 4K and 2K processing queues in sequence, but directly jumps to the 1K processing queue to obtain the pre-compressed data block;

[0031] S323. After jumping to the queue to be scanned and processed, the coroutine first determines whether there are pre-compressed data blocks in the current processing queue, and then performs the following operations according to different situations:

[0032] 1) If there are pre-compressed data blocks in the current processing queue, the coroutine scans the current processing queue, retrieves the pre-compressed data blocks, and performs a compaction and arrangement operation.

[0033] 2) If there are no pre-compressed data blocks in the current processing queue, but there are pre-compressed data blocks in the cache queue corresponding to the current processing queue, the coroutine obtains the pre-compressed data blocks in the corresponding cache queue and performs a compaction and arrangement operation. Considering the performance impact, the coroutine first locks the corresponding cache queue, then swaps the corresponding cache queue with the processing queue as a whole, and finally obtains all the pre-compressed data blocks in the corresponding cache queue and performs a compaction and arrangement operation.

[0034] 3) If there are no pre-compressed data blocks in the current processing queue and its corresponding cache queue, the coroutine skips the current processing queue and continues to scan the subsequent processing queues until it obtains an available pre-compressed data block. The coroutine then performs a compaction and arrangement operation on the obtained pre-compressed data block.

[0035] Furthermore, according to some embodiments of the present invention, the compressed data block in step S3 of the distributed storage data compression method of the present invention is composed of multiple sectors, each of size 512B, and each sector includes a sector header and N closely arranged pre-compressed data blocks.

[0036] The sector header includes a magic word, the number N of data blocks in the sector, and N metadata information equal to the number of pre-compressed data blocks in the sector; wherein:

[0037] The magic word is located at the beginning of each sector and is used to verify the validity of that sector;

[0038] The number of sector data blocks N is used to characterize the number of pre-compressed data blocks in the current sector;

[0039] Metadata is used to store the relevant attributes of pre-compressed data blocks within the current sector. The arrangement order of the N metadata items is consistent with the arrangement order of the pre-compressed data blocks within the current sector. The metadata consists of pre, next, and offset. Considering that a single pre-compressed data block may be arranged across sectors, the metadata is as follows:

[0040] "pre" indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors forward;

[0041] The next indicator indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors to the next;

[0042] offset represents the offset of the corresponding pre-compressed data block within the current sector.

[0043] Because excessive compaction can lead to too much metadata information, thus reducing the benefits of compaction, the compaction process introduces zero-padding logic, including zero-padding of sectors and zero-padding of compacted data blocks.

[0044] Furthermore, according to some embodiments of the present invention, step S3 of the above-described distributed storage data compression method, which involves compacting and arranging the pre-compressed data blocks to obtain compressed data blocks, further includes introducing zero-padding during the compaction and arrangement of the pre-compressed data blocks. The zero-padding operation includes sector zero-padding and compressed data block zero-padding, wherein:

[0045] Sector zeroing refers to maintaining a dynamically configured zeroing interval at the end of each sector (512B). After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zeroing interval of the current sector based on the remaining space of the current sector. If so, sector zeroing is performed, and then the coroutine will continue to compress and arrange the remaining pre-compressed data blocks starting from the next sector.

[0046] Zero padding for compressed data blocks refers to maintaining a dynamically configured zero padding range at the end of a compressed data block (8K). After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zero padding range of the current compressed data block based on the remaining space of the current compressed data block. If so, the zero padding operation is performed on the compressed data block, and the compressed arrangement process ends.

[0047] To reduce the impact on business latency, coroutines cannot perform compaction and arrangement operations for extended periods. Therefore, the compaction and arrangement process introduces timeout zero-filling logic.

[0048] Furthermore, according to some embodiments of the present invention, step S3 of the above-mentioned distributed storage data compression method, which involves compressing and arranging pre-compressed data blocks to obtain compressed data blocks, further includes introducing a timeout zero-padding operation during the compression and arrangement of pre-compressed data blocks. The timeout zero-padding operation means that the goroutine defaults to compressing and arranging pre-compressed data blocks into 8K compressed data blocks. However, if, for some reason, the goroutine fails to find enough available pre-compressed data blocks and the size of the compressed data block cannot reach 8K for a long time, and the current compression and arrangement process takes more than a preset threshold, considering the performance impact, even if the current tail address of the compressed data block does not enter the zero-padding interval of the compressed data block, the area of ​​the compressed data block that has not reached 8K will be directly padded with zeros. This timeout threshold supports dynamic configuration.

[0049] Secondly, the present invention also provides a distributed storage data compression device, the device comprising:

[0050] Precompression module: Used to precompress the original data blocks before data is written to disk, resulting in precompressed data blocks;

[0051] Profitability determination module: used to calculate the pre-compression profitability of pre-compressed data blocks and compare it with a preset threshold to determine whether the pre-compressed data block should enter the compaction and arrangement process;

[0052] Compressed data block layout module: used to compress and arrange pre-compressed data blocks to obtain compressed data blocks;

[0053] Zero-filling operation module: used to perform sector zero-filling, compacted data block zero-filling, and compacted data block timeout zero-filling operations.

[0054] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described distributed storage data compression method.

[0055] In summary, compared to current compression schemes in the field of distributed storage, the present invention provides a detailed overview of the data compression process, analyzes potential optimization points in the data compression process, and overcomes the defects and shortcomings of current compression schemes in the field of distributed storage. Specifically, the data compression method of the present invention has the following characteristics:

[0056] (1) Improved data compression rate

[0057] The method of this invention divides the compression process of the original data block into two parts: pre-compression and compaction arrangement. Based on the pre-compression, the data block is further compacted and arranged, which greatly improves the data compression rate.

[0058] (2) Balanced performance loss with data compression rate

[0059] This method optimizes the performance degradation points in the compression process while ensuring the data compression ratio:

[0060] For the pre-compression stage, the performance optimization points are as follows:

[0061] 1) When hardware supports it, use a hardware pre-compression strategy to reduce CPU load.

[0062] 2) When the pre-compression benefit of the original data block is less than the set threshold, the original data block is not compressed to prevent the additional metadata brought by the compression operation from occupying more space, and also to avoid the performance loss caused by decompressing data when reading compressed data.

[0063] For the compaction and arrangement stage, the performance optimization points are as follows:

[0064] 1) The system dynamically starts and stops coroutine tasks based on the amount of data to be processed in the cache queue, so as to avoid excessive impact on system performance from too many coroutine tasks.

[0065] 2) Create an exclusive processing queue for each coroutine that corresponds one-to-one with the cache queue to avoid mutual exclusion and preemption of processing queues between coroutines, which would affect performance;

[0066] 3) When retrieving pre-compressed data blocks from the cache queue, the coroutine only locks the corresponding cache queue and then swaps the corresponding cache queue with the processing queue as a whole, avoiding unnecessary impact on system performance due to an excessively large lock range;

[0067] 4) Introduce zero-padding logic for timeout of compressed data blocks to reduce the impact of the compressed layout process on business latency.

[0068] (3) This method has a simple architecture, is easy to implement, and has high security and reliability, which helps to improve the performance of distributed storage systems. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention, and not all embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a schematic diagram of the design architecture of the distributed storage data compression method of the present invention.

[0071] Figure 2 This is a schematic diagram of the design architecture of the compression and arrangement process in the data compression method of the present invention.

[0072] Figure 3 This is a schematic diagram of the internal format of the compressed data block in the data compression method of the present invention.

[0073] Figure 4 This is a schematic diagram of the sector zero-padding operation in the data compression and layout process of the present invention.

[0074] Figure 5 This is a schematic diagram of the zero-padding operation for compressed data blocks in the data compression and layout process of the present invention.

[0075] Figure 6 This is a schematic diagram of the zero-filling operation for timed-out data blocks in the data compression and layout process of the present invention.

[0076] Figure 7 This is a flowchart illustrating the overall implementation of the distributed storage data compression method of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. This invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this invention.

[0078] It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0079] It should be noted that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement equipment and / or practices.

[0080] the term:

[0081] Coroutines are a more lightweight concept than threads. They are neither processes nor threads; they are special functions that can be suspended at a certain point and can resume execution from where they were suspended.

[0082] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings.

[0083] The overall process of this solution is as follows: Before the data is pushed to disk, the original data blocks first enter the pre-compression process. The original data blocks are pre-compressed into pre-compressed data blocks. Subsequently, the pre-compressed data blocks are further compressed and arranged. Finally, multiple original data blocks are arranged together, thereby reducing the space occupied by the original data blocks.

[0084] according to Figure 1 and Figure 7 As shown, the distributed storage data compression method of the present invention includes:

[0085] S1. Before the data is written to disk, the original data blocks are pre-compressed to obtain pre-compressed data blocks;

[0086] For the original data blocks, the precompression scheme is as follows: Figure 1 As shown:

[0087] Figure 1In the process, the original data blocks are first pre-compressed. The pre-compressed data blocks include a compression header, data segments, and the key corresponding to the original data blocks. The compression header contains information such as the compression algorithm used during the pre-compression of the original data blocks, the data segment length, and the key length. For the pre-compression process, if a hardware acceleration card is detected in the storage system, a hardware pre-compression process is performed; otherwise, a software pre-compression process is performed. The results of software and hardware pre-compression are compatible with each other.

[0088] S2. Calculate the pre-compression gain of the pre-compressed data block obtained in the previous step and compare it with a preset threshold. If the pre-compression gain is greater than or equal to the preset threshold, the pre-compressed data block will continue to enter the next step of the compaction and arrangement process.

[0089] After the original data block is pre-compressed, the length of the original data block is subtracted from the length of the corresponding pre-compressed data block to obtain the pre-compression gain of the pre-compressed data block. If the pre-compression gain is greater than or equal to a preset threshold, the pre-compressed data block continues to enter the compaction and arrangement process. If the pre-compression gain is less than the preset threshold, it means that the additional metadata brought by the pre-compression operation will occupy more space, so the current pre-compression process needs to be terminated, and the original data block is not compressed and is directly removed from disk. Considering scalability, the threshold for pre-compression gain can be dynamically configured.

[0090] S3. Compact and arrange the pre-compressed data blocks to obtain compressed data blocks;

[0091] The overall plan for compaction and arrangement is as follows: Figure 2 As shown:

[0092] S31. Figure 2 In the process, pre-compressed data blocks entering the compaction and arrangement process are placed into corresponding cache queues according to their size. There are 10 cache queues, which are used to cache pre-compressed data blocks with sizes less than or equal to 8K, 4K, 2K, 1K, 512B, 256B, 128B, 64B, 32B, and 16B, respectively. For example, if the size of the pre-compressed data block is 6K, it will enter the 8K cache queue, and if the size of the pre-compressed data block is 800B, it will enter the 1K cache queue.

[0093] S32. After the pre-compressed data block enters the cache queue, the system starts a coroutine. This coroutine retrieves the pre-compressed data block from the cache queue and puts it into the processing queue. Then, it retrieves the pre-compressed data block from the processing queue and compresses and arranges it.

[0094] Since multiple coroutines may exist in the system at the same time, in order to avoid performance impact due to mutual exclusion and preemption of processing queues between coroutines, each coroutine has its own exclusive processing queue. The processing queue corresponds one-to-one with the cache queue, and there are 10 in total, which store pre-compressed data blocks from different cache queues.

[0095] The steps for compressing and arranging pre-compressed data blocks are as follows:

[0096] S321. The coroutine scans all processing queues in a loop, following the order from the 8K processing queue to the 16B processing queue.

[0097] S322. Based on the remaining space size of the compressed data block, the coroutine retrieves the next processing queue to be scanned that is less than or equal to the remaining space size, and then directly jumps to that processing queue to obtain the pre-compressed data block for compaction and arrangement. The direct jump method can skip processing queues that do not meet the conditions, reducing wasted work; for example, if the initial remaining space size of the compressed data block is 8K, the coroutine jumps to the 8K processing queue to obtain the pre-compressed data block. After all the pre-compressed data blocks in the 8K processing queue are compacted and arranged into the compressed data block, the remaining space size of the compressed data block is reduced to 1000B. At this time, the coroutine does not need to scan the 4K and 2K processing queues in sequence, but directly jumps to the 1K processing queue to obtain the pre-compressed data block;

[0098] S323. After jumping to the queue to be scanned and processed, the coroutine first determines whether there are pre-compressed data blocks in the current processing queue, and then performs the following operations according to different situations:

[0099] 1) If there are pre-compressed data blocks in the current processing queue, the coroutine scans the current processing queue, retrieves the pre-compressed data blocks, and performs a compaction and arrangement operation.

[0100] 2) If there are no pre-compressed data blocks in the current processing queue, but there are pre-compressed data blocks in the cache queue corresponding to the current processing queue, the coroutine obtains the pre-compressed data blocks in the corresponding cache queue and performs a compaction and arrangement operation. Considering the performance impact, the coroutine first locks the corresponding cache queue, then swaps the corresponding cache queue with the processing queue as a whole, and finally obtains all the pre-compressed data blocks in the corresponding cache queue and performs a compaction and arrangement operation.

[0101] 3) If there are no pre-compressed data blocks in the current processing queue and its corresponding cache queue, the coroutine skips the current processing queue and continues to scan the subsequent processing queues until it obtains an available pre-compressed data block. The coroutine then performs a compaction and arrangement operation on the obtained pre-compressed data block.

[0102] S33. When the amount of pre-compressed data blocks to be processed in the cache queue reaches a preset threshold, the system starts a new coroutine to quickly process the pre-compressed data blocks backlogged in the cache queue. At the same time, considering that too many coroutine tasks will impact system performance, the system needs to control the upper limit of the number of coroutines, which can be dynamically configured. When the amount of pre-compressed data blocks to be processed in the cache queue decreases to below the preset threshold, and this state continues for a period of time (the threshold for the duration can be dynamically configured), the system stops the corresponding number of coroutines to reduce the impact of too many coroutine tasks on system performance.

[0103] The internal format of the compressed data block is as follows: Figure 3 As shown:

[0104] Depend on Figure 3 It can be seen that the compressed data block consists of multiple sectors of equal size (512B), each sector including a sector header and N closely arranged pre-compressed data blocks;

[0105] The sector header includes a magic word, the number of data blocks N in the sector, and N metadata information equal to the number of pre-compressed data blocks within the sector; where:

[0106] The magic word is located at the beginning of each sector and is used to verify the validity of that sector;

[0107] The number of sector data blocks N is used to characterize the number of pre-compressed data blocks in the current sector;

[0108] Metadata is used to store the relevant attributes of pre-compressed data blocks within the current sector. The arrangement order of the N metadata items is consistent with the arrangement order of the pre-compressed data blocks within the current sector. The metadata consists of pre, next, and offset. Considering that a single pre-compressed data block may be arranged across sectors, the metadata is as follows:

[0109] "pre" indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors forward;

[0110] The next indicator indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors to the next;

[0111] offset represents the offset of the corresponding pre-compressed data block within the current sector.

[0112] Because excessive compaction can generate too much metadata, thus reducing its benefits, the compaction process incorporates zero-padding logic, including sector zero-padding and compacted data block zero-padding. Specific solutions are as follows: Figure 4 and Figure 5 As shown:

[0113] Figure 4In this context, sector zeroing refers to maintaining a dynamically configured zeroing interval at the end of each sector (512B). After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zeroing interval of the current sector based on the remaining space size of the current sector. If so, sector zeroing is performed, and then the coroutine will continue to compress and arrange the remaining pre-compressed data blocks starting from the next sector.

[0114] Figure 5 In this context, zero padding for compressed data blocks refers to maintaining a dynamically configured zero padding interval at the end of the compressed data block (8K). After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zero padding interval of the current compressed data block based on the remaining space of the current compressed data block. If so, the zero padding operation is performed on the compressed data block, and the compressed arrangement process ends.

[0115] To minimize the impact on business latency, coroutines cannot perform compaction and scheduling operations for extended periods. Therefore, the compaction and scheduling process incorporates timeout padding logic, as detailed below. Figure 6 As shown:

[0116] Figure 6 In this context, the timeout zero-padding operation refers to the coroutine's default arrangement of pre-compressed data blocks into 8K compacted data blocks. However, if, for some reason, the coroutine fails to find enough available pre-compressed data blocks, and the compacted data block size cannot reach 8K for a long time, and the current compaction and arrangement process takes more than a preset threshold, then, considering the performance impact, even if the current tail address of the compacted data block has not entered the zero-padding range of the compacted data block, the area of ​​the compacted data block that has not reached 8K will be directly padded with zeros. This timeout threshold supports dynamic configuration.

[0117] The present invention also provides a distributed storage data compression device, the device comprising:

[0118] Precompression module: Used to precompress the original data blocks before data is written to disk, resulting in precompressed data blocks;

[0119] Profitability determination module: used to calculate the pre-compression profitability of pre-compressed data blocks and compare it with a preset threshold to determine whether the pre-compressed data block should enter the compaction and arrangement process;

[0120] Compressed data block layout module: used to compress and arrange pre-compressed data blocks to obtain compressed data blocks;

[0121] Zero-filling operation module: used to perform sector zero-filling, compacted data block zero-filling, and compacted data block timeout zero-filling operations.

[0122] Each module operates according to the distributed storage data compression method described above.

[0123] The various embodiments in this invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other.

[0124] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, substitutions, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A distributed storage data compression method, characterized in that, The method includes: S1. Before the data is written to disk, the original data block is pre-compressed to obtain a pre-compressed data block; the pre-compressed data block includes a compression header, a data segment, and a key corresponding to the original data block; The compression header contains the compression algorithm used during the pre-compression of the original data block, the data segment length, and the key length information. S2. Calculate the pre-compression gain of the pre-compressed data block obtained in the previous step and compare it with a preset threshold. If the pre-compression gain is greater than or equal to the preset threshold, the pre-compressed data block continues to enter the next compaction and arrangement process. The pre-compression gain is the difference between the length of the original data block and the length of the corresponding pre-compressed data block. If the pre-compression gain is less than the preset threshold, the current pre-compression process is terminated, and the original data block is not compressed and is directly removed from the disk. S3. Compact and arrange the pre-compressed data blocks to obtain compacted data blocks, including: S31. Pre-compressed data blocks entering the compaction and arrangement process are placed into buffer queues of corresponding sizes according to their different sizes; S32. After the pre-compressed data block enters the cache queue, the system starts a coroutine. This coroutine retrieves the pre-compressed data block from the cache queue and puts it into the processing queue. Then, it retrieves the pre-compressed data block from the processing queue and arranges it in a compact manner. Each coroutine has multiple exclusive processing queues, which correspond one-to-one with the cache queue. The multiple processing queues store pre-compressed data blocks from different cache queues. When retrieving a pre-compressed data block from the cache queue, the coroutine only locks the corresponding cache queue and then swaps the corresponding cache queue with the processing queue as a whole. S33. When the amount of pre-compressed data blocks to be processed in the cache queue reaches a preset threshold, the system starts a new coroutine. When the amount of pre-compressed data blocks to be processed in the cache queue decreases to below the preset threshold, the system stops the corresponding number of coroutines.

2. The distributed storage data compression method according to claim 1, characterized in that, The pre-compression process described in step S1 is as follows: 1) If a hardware acceleration card is detected in the storage system, a hardware pre-compression process is performed; 2) If no hardware acceleration card is detected in the storage system, proceed with the software pre-compression process; 3) The software and hardware pre-compression results are compatible with each other.

3. The distributed storage data compression method according to claim 1, characterized in that, The cache queues mentioned in step S31 consist of 10 queues, which are used to cache pre-compressed data blocks with sizes less than or equal to 8K, 4K, 2K, 1K, 512B, 256B, 128B, 64B, 32B, and 16B, respectively. In step S33, when the amount of data in the pre-compressed data blocks to be processed in the cache queue reaches a preset threshold, the system starts a new coroutine. Each coroutine has its own exclusive processing queue, and there are a total of 10 processing queues.

4. The distributed storage data compression method according to claim 3, characterized in that, The steps for compressing and arranging pre-compressed data blocks are as follows: S321. The coroutine scans all processing queues in a loop, following the order from the 8K processing queue to the 16B processing queue. S322. Based on the remaining space size of the compressed data block, the coroutine retrieves the next queue to be scanned that is less than or equal to the remaining space size, and then jumps directly to that queue. S323. After navigating to the queue to be scanned, the coroutine first determines whether a pre-compressed data block exists in the current processing queue, and then performs the following operations depending on the situation: 1) If there are pre-compressed data blocks in the current processing queue, the coroutine scans the current processing queue, retrieves the pre-compressed data blocks, and performs a compaction and arrangement operation. 2) If there are no pre-compressed data blocks in the current processing queue, but there are pre-compressed data blocks in the corresponding cache queue, the coroutine will retrieve the pre-compressed data blocks from the corresponding cache queue and perform a compaction and arrangement operation. 3) If there are no pre-compressed data blocks in the current processing queue and its corresponding cache queue, the coroutine skips the current processing queue and continues to scan the subsequent processing queues until it obtains an available pre-compressed data block. The coroutine then performs a compaction and arrangement operation on the obtained pre-compressed data block.

5. The distributed storage data compression method according to claim 1, characterized in that, The compressed data block mentioned in step S3 consists of multiple sectors, each 512B in size, and each sector includes a sector header and N closely arranged pre-compressed data blocks. The sector header includes a magic word, the number N of data blocks in the sector, and N metadata information equal to the number of pre-compressed data blocks in the sector; wherein: The magic word is located at the beginning of each sector and is used to verify the validity of that sector; The number of sector data blocks N is used to characterize the number of pre-compressed data blocks in the current sector; Metadata is used to store the relevant attributes of pre-compressed data blocks within the current sector. The arrangement order of the N metadata items is consistent with the arrangement order of the pre-compressed data blocks within the current sector. The metadata consists of pre, next, and offset, where: "pre" indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors forward; The next indicator indicates whether the corresponding pre-compressed data block in the current sector is arranged across sectors to the next; offset represents the offset of the corresponding pre-compressed data block within the current sector.

6. The distributed storage data compression method according to claim 5, characterized in that, Step S3, which involves compacting and arranging the pre-compressed data blocks to obtain compacted data blocks, further includes introducing zero-padding during the compaction and arrangement process. This zero-padding operation includes sector zero-padding and compacted data block zero-padding, wherein: Sector zeroing refers to maintaining a dynamically configured zeroing interval at the end of each sector. After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zeroing interval of the current sector based on the remaining space in the current sector. If so, sector zeroing is performed, and then the coroutine will continue to compress and arrange the remaining pre-compressed data blocks starting from the next sector. Zero padding for compressed data blocks refers to maintaining a dynamically configured zero padding range at the end of the compressed data block. After a single pre-compressed data block is arranged, the system determines whether the current tail address of the compressed data block enters the zero padding range of the current compressed data block based on the remaining space of the current compressed data block. If so, the zero padding operation is performed on the compressed data block, and the compression and arrangement process ends.

7. The distributed storage data compression method according to claim 6, characterized in that, Step S3, which involves compacting and arranging the pre-compressed data blocks to obtain compacted data blocks, also includes introducing a timeout zero-padding operation during the compaction and arrangement process. The timeout zero-padding operation means that the goroutine defaults to compacting and arranging the pre-compressed data blocks into 8K compacted data blocks. However, if the goroutine fails to find enough available pre-compressed data blocks and the compacted data block size cannot reach 8K, and the current compaction and arrangement process takes more than a preset threshold time, even if the current tail address of the compacted data block does not enter the zero-padding interval of the compacted data block, the area of ​​the compacted data block that does not reach 8K will be directly padded with zeros.

Images