ZSTD decompression method, device, equipment and medium

Abstract

The invention discloses a ZSTD decompression method and device, equipment and a medium, and relates to the technical field of data decompression, and the method comprises the steps: initializing a compression frame header, carrying out the partitioning of original data, so as to obtain a plurality of data blocks, and carrying out the ZSTD compression of uncompressed data blocks in the plurality of data blocks, so as to obtain compressed blocks; updating a target field in the compression frame header based on the compression block, skipping to the step of performing ZSTD compression on uncompressed data blocks in the plurality of data blocks until the plurality of data blocks are all compressed, obtaining a compression frame comprising the compression frame header, and storing the compression frame in a disk file; when the compressed frame is decompressed, analyzing a compressed frame header of the compressed frame so as to determine the total length of the compressed data according to the target field; and determining a data reading range based on the total length of the compressed data, reading corresponding target data from the disk file according to the data reading range, and performing ZSTD decompression on the target data to obtain original data.

Description

Technical Field

[0001] This invention relates to the field of data decompression technology, and in particular to a ZSTD decompression method, apparatus, equipment and medium. Background Technology

[0002] In existing ZSTD (Zstandard, a lossless data compression algorithm) compression technology, the frame header only records the original data size. During decompression, after reading the frame header, each block header needs to be parsed one by one, and then the block data needs to be read until the end block that marks the end of a single frame of compressed data is identified. This single-read operation requires estimation based on the original data size, resulting in redundant disk data reads and increased I / O (Input / Output) overhead. Streaming decompression requires reading the block header and block data one by one, leading to multiple I / O operations and reduced I / O performance. Relying on the end block to determine the end position of compressed data increases disk read volume, especially in scenarios with large disk sectors and small compressed blocks, where the redundant read problem is more pronounced.

[0003] In summary, optimizing the ZSTD decompression method to reduce disk reads and improve decompression efficiency is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a ZSTD decompression method, apparatus, device, and medium that can optimize the ZSTD decompression method to reduce disk read operations and improve decompression efficiency. The specific solution is as follows:

[0005] Firstly, this application provides a ZSTD decompression method, including:

[0006] The compressed frame header is initialized, and the original data is divided into blocks to obtain several corresponding data blocks. The uncompressed data blocks in the several data blocks are ZSTD compressed to obtain corresponding compressed blocks. The compressed frame header includes a target field that characterizes the total length of the compressed data.

[0007] Update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file;

[0008] When decompressing the compressed frame, the compressed frame header of the compressed frame is parsed to determine the total length of the compressed data based on the obtained target field;

[0009] The range of data to be read is determined based on the total length of the compressed data, and the corresponding target data is read from the disk file according to the range of data to be read, and the target data is decompressed by ZSTD to obtain the original data.

[0010] Optionally, before initializing the compressed frame header, the method further includes:

[0011] The target descriptor field in several frame header descriptors of the compressed frame header is determined; the target descriptor field includes a descriptor field representing currently unused bits and a descriptor field representing reserved bits.

[0012] Change the flag bit of the target descriptor field to a new flag bit that represents the total length of the compressed data, so as to add the target field to the frame header structure of the compressed frame header, and adjust the ZSTD compression end source code and decompression end source code accordingly.

[0013] The rules used for the newly added flag bits are consistent with the rules used for the flag bits that characterize the length of the original data before compression; the length of the target field is determined based on the newly added flag bits.

[0014] Optionally, performing ZSTD compression on the uncompressed data blocks among the plurality of data blocks to obtain corresponding compressed blocks includes:

[0015] ZSTD compression is performed on the uncompressed data blocks in the plurality of data blocks to obtain the corresponding compression result; the compression result includes compressed block data;

[0016] Based on the compression result, a corresponding compressed block header is generated to obtain a compressed block including the compressed block header and the compressed block data;

[0017] The compressed block header is saved to a preset compressed block header buffer, and the compressed block data is saved to a preset compressed block data buffer.

[0018] Optionally, updating the target field in the compressed frame header based on the compressed block includes:

[0019] The compressed block header is obtained and parsed from the compressed block header buffer to determine the compressed block length, and the compressed block length is added to the target field in the compressed frame header to generate the target compressed frame header;

[0020] Accordingly, obtaining the compressed frame including the compressed frame header includes:

[0021] Data encapsulation is performed based on the target compressed frame header, the compressed block header in the compressed block header buffer, and the compressed block data in the compressed block data buffer to obtain the compressed frame corresponding to the compressed original data.

[0022] Optionally, parsing the compressed frame header of the compressed frame to determine the total length of the compressed data based on the obtained target field includes:

[0023] Read the compressed frame header of the compressed frame and obtain the frame identifier in the compressed frame header. Verify the format validity of the compressed frame header based on the frame identifier. After the verification is successful, parse the frame header descriptor of the compressed frame header and determine the corresponding descriptor flag bit.

[0024] Based on the descriptor flags, the unparsed structure of the compressed frame header is parsed to determine the total length of the compressed data according to the obtained target field, and to determine the original data length of the original data.

[0025] Optionally, determining the data reading range based on the total length of the compressed data includes:

[0026] Determine the end position of the compressed frame header in the compressed frame, and determine the header length of the compressed frame header;

[0027] The sum of the end position, the frame header length, and the total length of the compressed data is determined, and the end position is determined as the starting position of the data reading range, and the sum is determined as the ending position of the data reading range.

[0028] Optionally, the step of performing ZSTD decompression on the target data to obtain the original data includes:

[0029] The target data is saved to a preset decompression buffer, and the compressed block headers of several compressed blocks are read sequentially from the decompression buffer;

[0030] Parse the compressed block header to obtain the corresponding compressed block data length and compression type;

[0031] According to the length of the compressed block data, the corresponding compressed block data is extracted from the decompression buffer, and the compressed block data is decompressed according to the compression type, so as to save the decompressed data to the preset original data buffer.

[0032] The decompressed data in the original data buffer is merged to obtain merged decompressed data, and the length of the decompressed data of the merged decompressed data is determined.

[0033] The length of the decompressed data is compared with the length of the original data. If the length of the decompressed data is the same as the length of the original data, the decompression is determined to be successful, and the merged decompressed data is identified as the original data.

[0034] Secondly, this application provides a ZSTD decompression apparatus, comprising:

[0035] The data block compression module is used to initialize the compression frame header, divide the original data into blocks to obtain several corresponding data blocks, and perform ZSTD compression on the uncompressed data blocks in the several data blocks to obtain corresponding compressed blocks; the compression frame header includes a target field characterizing the total length of the compressed data.

[0036] The compressed frame saving module is used to update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file.

[0037] The frame header parsing module is used to parse the compressed frame header of the compressed frame when decompressing the compressed frame, so as to determine the total length of the compressed data according to the obtained target field;

[0038] The data decompression module is used to determine the data reading range based on the total length of the compressed data, read the corresponding target data from the disk file according to the data reading range, and perform ZSTD decompression on the target data to obtain the original data.

[0039] Thirdly, this application provides an electronic device, comprising:

[0040] Memory, used to store computer programs;

[0041] A processor is used to execute the computer program to implement the aforementioned ZSTD decompression method.

[0042] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned ZSTD decompression method.

[0043] In this application, a compressed frame header is initialized, and the original data is divided into blocks to obtain several corresponding data blocks. Uncompressed data blocks within these blocks are then ZSTD compressed to obtain compressed blocks. The compressed frame header includes a target field representing the total length of the compressed data. Based on the compressed blocks, the target field in the compressed frame header is updated, and the process jumps to the step of ZSTD compression of the uncompressed data blocks until all data blocks are compressed, resulting in a compressed frame including the compressed frame header. The compressed frame is then saved to a disk file. When decompressing the compressed frame, the compressed frame header is parsed to determine the total length of the compressed data based on the obtained target field. A data reading range is determined based on the total length of the compressed data, and corresponding target data is read from the disk file according to the data reading range. The target data is then ZSTD decompressed to obtain the original data. As can be seen from the above, this application first initializes a compressed frame header containing a target field representing the total length of the compressed data, divides the original data into several data blocks, performs ZSTD compression on the uncompressed data blocks to obtain compressed blocks, updates the target field of the compressed frame header based on the compressed blocks, and repeats the ZSTD compression steps for the uncompressed data blocks until all data blocks are compressed, forming a compressed frame containing the compressed frame header and saving it to a disk file. During decompression, the compressed frame header is first parsed, the total length of the compressed data is determined based on the target field, the range of data to be read is determined based on the total length of the compressed data, the corresponding target data is read from the disk file and ZSTD decompression is performed, and finally the original data is obtained. In this way, through the process described above in this application, block compression and dynamic updating of the frame header target field can ensure that the compressed frame header information accurately matches the actual compressed data; during decompression, the data range can be quickly located through the frame header, eliminating the need to read redundant data, reducing disk read volume or number of reads, reducing IO overhead, making the ZSTD compression and decompression process both efficient and accurate, ensuring the integrity of data compression, improving the efficiency of data retrieval and recovery during decompression, and thus optimizing the ZSTD decompression method to reduce disk read volume and improve decompression efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This is a flowchart of a ZSTD decompression method disclosed in this application;

[0046] Figure 2 This is a schematic diagram of an improved ZSTD compression process disclosed in this application;

[0047] Figure 3 This is a schematic diagram of an improved process for a ZSTD decompression terminal disclosed in this application;

[0048] Figure 4 This is a schematic diagram of the structure of a ZSTD decompression device disclosed in this application;

[0049] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In existing ZSTD compression technology, the frame header only records the original data size. During decompression, after reading the frame header, each block header needs to be parsed one by one, and then the block data needs to be read until the end block that marks the end of a single frame of compressed data is identified. This results in redundant disk reads and increased I / O overhead when reading data in a single pass, as it requires estimation based on the original data size. Streaming decompression requires reading the block header and block data one by one, leading to multiple I / O operations and reduced I / O performance. Relying on the end block to determine the end position of compressed data increases disk read volume, especially in scenarios with large disk sectors and small compressed blocks, where the problem of redundant reads is more pronounced.

[0052] To overcome the aforementioned technical problems, this application provides a ZSTD decompression method that optimizes the ZSTD decompression method to reduce disk read volume and improve decompression efficiency.

[0053] See Figure 1 As shown, this embodiment of the invention discloses a ZSTD decompression method, including:

[0054] Step S11: Initialize the compressed frame header, divide the original data into blocks to obtain several corresponding data blocks, and perform ZSTD compression on the uncompressed data blocks in the several data blocks to obtain corresponding compressed blocks; the compressed frame header includes a target field that characterizes the total length of the compressed data.

[0055] In this embodiment, a compressed frame header containing a target field (Compressed Size) representing the total length of the compressed data is initialized. This target field is then set to zero. The original data is divided into several independent data blocks according to a preset block size (configurable, such as 16KB or 32KB). If the original data is less than one block size, a short block is generated. The uncompressed data blocks within this short block are then compressed using the ZSTD algorithm to obtain the corresponding compressed block. The target field is a newly added frame header field in this application, used to store the total length of a single frame of ZSTD compressed data (including the total number of bytes of all compressed block headers and compressed block data). The field type is a 64-bit unsigned integer, supporting a maximum compressed data size of 2^64 bytes. Figure 2 The diagram shown is a schematic of an improved process for a ZSTD compression terminal provided in this application.

[0056] It should be noted that before initializing the compressed frame header, the frame header structure of the compressed frame header needs to be optimized and the decompression process improved. The processing flow is as follows: The target descriptor field in several frame header descriptors of the compressed frame header is determined; the target descriptor field includes a descriptor field representing currently unused data and a descriptor field representing reserved bits; the flag bits of the target descriptor field are changed to new flag bits representing the total length of the compressed data, so as to add the target field to the frame header structure of the compressed frame header, and the ZSTD compression and decompression source code are adjusted accordingly; wherein, the rule adopted by the new flag bits is consistent with the rule adopted by the flag bits representing the original data length before compression; the length of the target field is determined based on the new flag bits. That is, Table 1 below shows a schematic table of the frame header structure of an existing ZSTD frame header provided in this application, and Table 2 shows a schematic table of the structure of an existing ZSTD frame header descriptor provided in this application.

[0057] Table 1. Schematic diagram of existing ZSTD frame header structure

[0058]

[0059] Table 2. Schematic diagram of the structure of existing ZSTD frame header descriptors

[0060]

[0061] In this design, the window descriptor, dictionary identifier, and frame content size are all optional, non-essential fields. From several frame header descriptors in the compressed frame header, a target descriptor field is determined, containing both currently unused and reserved descriptor fields. The flag of this target descriptor field is modified to a new flag (Compress_Size_Flag) representing the total length of the compressed data. This adds the target field (its length is 0-8 bytes, determined by the new flag) to the frame header structure of the compressed frame header. Correspondingly, the ZSTD compression source code (core files include `zstd_compress.c` and `zstd_frame.h`) and decompression source code (core files include `zstd_decompress.c` and `zstd_frame.h`) are adjusted. The new flag uses the same rules as the flag representing the original data length before compression (Frame_Content_Size_Flag). Table 3 below shows a schematic diagram of the modified ZSTD frame header structure provided in this application; Table 4 shows a schematic diagram of the modified ZSTD frame header descriptor structure provided in this application; and Table 5 shows a schematic diagram of the rules for flag bits representing the length of the original data before compression provided in this application.

[0062] Table 3. Schematic diagram of the modified ZSTD frame header structure

[0063]

[0064] Table 4. Schematic diagram of the modified ZSTD frame header descriptor

[0065]

[0066] Table 5. Schematic diagram of the rules for flag bits representing the length of the original data before compression.

[0067]

[0068] It should be noted that the process of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks to obtain the corresponding compressed blocks is as follows: ZSTD compression is performed on the uncompressed data blocks in the plurality of data blocks to obtain the corresponding compression result; the compression result includes compressed block data; a corresponding compressed block header is generated based on the compression result to obtain a compressed block including the compressed block header and the compressed block data; the compressed block header is saved to a preset compressed block header buffer, and the compressed block data is saved to a preset compressed block data buffer. That is, ZSTD compression (compression level is configurable) is performed on the uncompressed data blocks in the plurality of data blocks to obtain a compression result containing compressed block data; a corresponding compressed block header is generated based on the compression result, thereby forming a compressed block composed of the compressed block header and the compressed block data; then the compressed block header is stored in a preset compressed block header buffer (denoted as BlockHeaderBuffer), and the compressed block data is stored in a preset compressed block data buffer (denoted as BlockDataBuffer). It should be further noted that each compressed block header includes a Block Size field (used to record the number of bytes after compression of the data block, occupying 21 bits, supporting a maximum single-block compressed data of 2^21 bytes), a compression type (such as no compression, fast compression, high compression ratio compression), and an identifier field indicating whether it is the last block. In this way, this embodiment utilizes unused and reserved descriptor fields in the frame header to add target fields without reconstructing the overall structure of the compressed frame header. While maintaining compatibility with the original frame header design, it supplements crucial information on the total length of the compressed data, and the newly added flags follow existing rules, reducing the complexity of source code adjustments and improving the data adaptability and reliability of the ZSTD compression and decompression process. Block compression improves the flexibility of data processing, avoids efficiency losses caused by processing large amounts of data at once, and ensures the orderly progress of ZSTD compression. Compressing data first, then generating compressed block headers, and finally storing them in separate buffers makes the structure of the compressed blocks clearer and more organized. Buffer storage avoids confusion between the block header and the data, providing convenient retrieval and calling conditions for subsequent compressed frame header updates and data integration, improving the efficiency and accuracy of subsequent data processing.

[0069] Step S12: Update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file.

[0070] In this embodiment, the target field representing the total length of the compressed data in the compressed frame header is updated according to the compressed block. Then, the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks is returned. This process is repeated until all data blocks are compressed, and finally a compressed frame containing the compressed frame header is formed and saved to a disk file.

[0071] It should be noted that the processing flow for updating the target field in the compressed frame header based on the compressed block is as follows: The compressed block header is obtained and parsed from the compressed block header buffer to determine the compressed block length, and the compressed block length is added to the target field in the compressed frame header to generate the target compressed frame header. That is, the compressed block header is obtained from the preset compressed block header buffer and parsed to determine the compressed block length of the corresponding compressed block. The compressed block length is added to the target field in the compressed frame header, and finally, the target compressed frame header is generated based on the various flag bits of the frame header descriptor. The compressed block length includes the block header length (3 bytes) and the block data length; the compressed frame header includes a magic number (0xFD2FB528, indicating ZSTD format), a frame header length (recording the total number of bytes in the FrameHeader), a Frame Content Size (recording the total number of bytes in the original data, supporting 32-bit or 64-bit), a window size (recording the window memory size to be allocated during decompression), and a checksum flag (indicating whether frame tail checksum is enabled). It should be further noted that the processing flow for obtaining the compressed frame including the compressed frame header is as follows: Data encapsulation is performed based on the target compressed frame header, the compressed block header in the compressed block header buffer, and the compressed block data in the compressed block data buffer to obtain the compressed frame corresponding to the compressed original data. That is, taking the target compressed frame header as the core, and combining the compressed block header stored in the compressed block header buffer and the compressed block data stored in the compressed block data buffer, unified data encapsulation processing is performed in the order of "target compressed frame header → several sets of compressed block headers + compressed block data → frame tail checksum," ensuring that all fields are aligned (meeting the byte alignment requirements of the ZSTD format to avoid parsing misalignment during decompression), ultimately obtaining the compressed frame corresponding to the original data. In this way, this embodiment processes all data blocks in batches and saves them uniformly as compressed frames, ensuring the integrity and standardization of ZSTD compression. It provides a reliable length basis for data verification and reading during subsequent decompression, improving the stability of compressed data storage and the convenience of subsequent use. By parsing the compressed block header in the buffer to obtain the length and gradually accumulating and updating the target field of the frame header, the total length of all compressed blocks can be accurately counted, helping to quickly locate the data range and verify data integrity during subsequent decompression. At the same time, relying on the buffer to quickly obtain the block header data improves the efficiency and accuracy of frame header updates. Integrating key frame headers, all block headers, and block data into a single encapsulation makes the structure of the compressed frame complete and regular, ensuring the integrity and standardization of compressed data and improving the continuity of compressed data storage and subsequent use.

[0072] Step S13: When decompressing the compressed frame, parse the compressed frame header of the compressed frame to determine the total length of the compressed data based on the obtained target field.

[0073] In this embodiment, when decompressing the compressed frame, the compressed frame header is first parsed, and the total length of the compressed data is determined through the target field in the header. For example... Figure 3 The diagram shown is a schematic of an improved process for a ZSTD decompression terminal provided in this application.

[0074] Specifically, the compressed frame header of the compressed frame is read, and the frame identifier in the compressed frame header is obtained. The format validity of the compressed frame header is verified based on the frame identifier. After successful verification, the frame header descriptor of the compressed frame header is parsed to determine the corresponding descriptor flag. Based on the descriptor flag, the unparsed structure of the compressed frame header is parsed to determine the total length of the compressed data and the original data length of the original data. That is, the compressed frame header of the compressed frame is read, and the frame identifier therein is obtained. The format validity of the compressed frame header is verified through the frame identifier. After successful verification, the frame header descriptor of the compressed frame header is parsed to determine the corresponding descriptor flag. Based on the descriptor flag, the unparsed structure of the compressed frame header is parsed, and then the total length of the compressed data is determined through the target field, while simultaneously determining the original data length of the original data. In addition, this embodiment can also determine the window size (allocate decompression window memory) and checksum flag (determine whether checksum is required) through the compressed frame header. In this way, this embodiment first parses the target field of the frame header before decompression, which can quickly obtain the core length information of the compressed data and improve the accuracy of data positioning during decompression. First verifying the legality of the frame header format and then parsing the structure step by step can eliminate compressed frames with abnormal formats, avoid invalid decompression operations, ensure the safety and standardization of the decompression process, and improve decompression efficiency and data restoration accuracy.

[0075] Step S14: Determine the data reading range based on the total length of the compressed data, read the corresponding target data from the disk file according to the data reading range, and perform ZSTD decompression on the target data to obtain the original data.

[0076] In this embodiment, the range of data to be read is determined based on the total length of the compressed data obtained from the parsing. The corresponding target data (including BlockHeaderBuffer and BlockDataBuffer) is read from the disk file according to the range. The ZSTD decompression operation is performed on the target data, and finally the original data is restored.

[0077] It should be noted that the processing flow for determining the read data range based on the total length of the compressed data is as follows: Determine the end position of the compressed frame header within the compressed frame, and determine the header length of the compressed frame header; determine the sum of the end position, the header length, and the total length of the compressed data, and determine the end position as the start position of the read data range and the sum as the end position of the read data range. That is, first determine the end position of the compressed frame header within the compressed frame, and the header length of the frame header itself; calculate the sum of the end position, the header length, and the total length of the compressed data; set the end position as the start position of the read data range; and set the sum as the end position of the read data range.

[0078] It should be further noted that the process of performing ZSTD decompression on the target data to obtain the original data is as follows: The target data is saved to a preset decompression buffer, and the headers of several compressed blocks are read sequentially from the decompression buffer; the compressed block headers are parsed to obtain the corresponding compressed block data length and compression type; the corresponding compressed block data is extracted from the decompression buffer according to the compressed block data length, and the compressed block data is decompressed based on the compression type to save the obtained decompressed data to a preset original data buffer; the decompressed data in the original data buffer is merged to obtain merged decompressed data, and the decompressed data length of the merged decompressed data is determined; the decompressed data length is compared with the original data length, and if the decompressed data length is consistent with the original data length, decompression is considered successful, and the merged decompressed data is identified as the original data. That is, the target data is stored in a preset decompression buffer (denoted as CompressedBuffer). The compressed block headers of several compressed blocks are read sequentially from the decompression buffer. After parsing, the corresponding compressed block data length and compression type are obtained. The compressed block data is extracted from the decompression buffer according to the compressed block data length. After decompression according to the compression type, it is stored in a preset original data buffer. The decompressed data in the original data buffer is merged. Since the CompressedBuffer already contains all valid data block data, there is no need to read data from the disk again, avoiding redundant I / O. Finally, the length of the merged decompressed data is determined and compared with the length of the original data to verify integrity. If the two are consistent, the decompression is considered successful, and the merged data is the original data. In this way, this embodiment accurately locks the reading range based on key information in the frame header, which can avoid the problem of reading redundant data or insufficient data during decompression, ensuring the efficiency of the decompression process and the integrity of the original data restoration. Combined with the ZSTD algorithm for efficient decompression, it ensures the integrity and accuracy of the original data restoration, while improving the overall efficiency of the decompression process. The length comparison verification step can effectively detect data loss or errors during the decompression process, improving the reliability of the decompression operation.

[0079] As can be seen from the above, the embodiments of this application first initialize a compressed frame header containing a target field representing the total length of the compressed data, divide the original data into several data blocks, perform ZSTD compression on the uncompressed data blocks to obtain compressed blocks, update the target field of the compressed frame header based on the compressed blocks, and repeat the ZSTD compression steps for the uncompressed data blocks until all data blocks are compressed, forming a compressed frame containing the compressed frame header and saving it to a disk file. During decompression, the compressed frame header is first parsed, the total length of the compressed data is determined according to the target field, and then the range of data to be read is determined based on the total length of the compressed data. The corresponding target data is read from the disk file and ZSTD decompression is performed to finally obtain the original data. In this way, through the above-described process of the embodiments of this application, block compression and dynamic updating of the target field of the frame header can ensure that the compressed frame header information accurately matches the actual compressed data; during decompression, the data range can be quickly located through the frame header, eliminating the need to read redundant data, reducing disk read volume or number of reads, reducing IO overhead, making the ZSTD compression and decompression process both efficient and accurate, ensuring the integrity of data compression, improving the efficiency of data retrieval and recovery during decompression, and thus optimizing the ZSTD decompression method to reduce disk read volume and improve decompression efficiency.

[0080] As can be seen from the previous embodiment, this application discloses a ZSTD decompression method that can optimize the ZSTD decompression method to reduce disk read volume and improve decompression efficiency. Next, taking "10MB of original data compressed into 3 blocks" as an example, the ZSTD decompression method will be described in detail.

[0081] The first step is compression. The original data is divided into blocks: 10MB of original data is divided into 3 blocks (Block_1=4MB, Block_2=3MB, Block_3=3MB); Block compression: Block_1 is compressed to S_1=2048KB, Block_2 is compressed to S_2=1843KB, and Block_3 is compressed to S_3=2252KB; The total number of bytes in the BlockHeader is 3 bytes × 3 = 9 bytes; The compressed size is 9 + (2048×1024 + 1843×1024 + 2252×1024) = 9 + 6143×1024 = 6290441 bytes (approximately 6MB); The compressed size is written to the Compressed Size field of the Frame Header and encapsulated into a compressed file.

[0082] Finally, during the decompression stage, the Frame Header is parsed to obtain a Compressed Size of 6290441 bytes; the effective data range is calculated as follows: StartPos = Frame Header length (assuming an original 20 bytes + an additional 8 bytes = 28 bytes) → EndPos = 28 + 6290441 = 6290469 bytes; only the effective data (approximately 6MB) from the disk (28~6290469 bytes) is read and stored in the CompressedBuffer; the three Block Headers and Block Data in the CompressedBuffer are parsed, and after decompression, they are merged into 10MB of original data, with no redundant disk reads.

[0083] Accordingly, see Figure 4 As shown in the illustration, this application also provides a ZSTD decompression apparatus, comprising:

[0084] The data block compression module 11 is used to initialize the compression frame header, divide the original data into blocks to obtain a number of corresponding data blocks, and perform ZSTD compression on the uncompressed data blocks in the number of data blocks to obtain corresponding compressed blocks; the compression frame header includes a target field that characterizes the total length of the compressed data.

[0085] The compressed frame saving module 12 is used to update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file.

[0086] The frame header parsing module 13 is used to parse the compressed frame header of the compressed frame when decompressing the compressed frame, so as to determine the total length of the compressed data according to the obtained target field;

[0087] The data decompression module 14 is used to determine the data reading range based on the total length of the compressed data, read the corresponding target data from the disk file according to the data reading range, and perform ZSTD decompression on the target data to obtain the original data.

[0088] In some specific embodiments, the ZSTD decompression device may further include:

[0089] A field determination unit is used to determine the target descriptor field among a plurality of frame header descriptors in the compressed frame header; the target descriptor field includes a descriptor field representing currently unused bits and a descriptor field representing reserved bits;

[0090] The flag bit changing unit is used to change the flag bit of the target descriptor field to a new flag bit that represents the total length of the compressed data, so as to add the target field in the frame header structure of the compressed frame header and adjust the compression end source code and decompression end source code of ZSTD accordingly.

[0091] The rules used for the newly added flag bits are consistent with the rules used for the flag bits that characterize the length of the original data before compression; the length of the target field is determined based on the newly added flag bits.

[0092] In some specific embodiments, the data block compression module 11 may specifically include:

[0093] A data block compression unit is used to perform ZSTD compression on uncompressed data blocks among the plurality of data blocks to obtain a corresponding compression result; the compression result includes compressed block data;

[0094] A compression block header generation unit is used to generate a corresponding compression block header based on the compression result, so as to obtain a compression block including the compression block header and the compression block data;

[0095] A data storage unit is used to save the compressed block header to a preset compressed block header buffer and the compressed block data to a preset compressed block data buffer.

[0096] In some specific embodiments, the compressed frame storage module 12 may specifically include:

[0097] The length accumulation unit is used to obtain and parse the compressed block header from the compressed block header buffer to determine the compressed block length, and accumulate the compressed block length to the target field in the compressed frame header to generate the target compressed frame header;

[0098] Accordingly, the compressed frame storage module 12 may specifically include:

[0099] The data encapsulation unit is used to encapsulate data based on the target compressed frame header, the compressed block header in the compressed block header buffer, and the compressed block data in the compressed block data buffer to obtain the compressed frame corresponding to the compressed original data.

[0100] In some specific embodiments, the frame header parsing module 13 may specifically include:

[0101] The descriptor parsing unit is used to read the compressed frame header of the compressed frame, obtain the frame identifier in the compressed frame header, verify the format legality of the compressed frame header according to the frame identifier, and after the verification is passed, parse the frame header descriptor of the compressed frame header to determine the corresponding descriptor flag bit.

[0102] The structure parsing unit is used to parse the unparsed structure of the compressed frame header based on the descriptor flag, so as to determine the total length of the compressed data according to the obtained target field, and to determine the original data length of the original data.

[0103] In some specific embodiments, the data decompression module 14 may specifically include:

[0104] A length determination unit is used to determine the end position of the compressed frame header in the compressed frame and to determine the header length of the compressed frame header;

[0105] The position determination unit is used to determine the sum of the end position, the frame header length, and the total length of the compressed data, and to determine the end position as the starting position of the data reading range and the sum as the ending position of the data reading range.

[0106] In some specific embodiments, the data decompression module 14 may specifically include:

[0107] A block header reading unit is used to save the target data to a preset decompression buffer and sequentially read the compressed block headers of several compressed blocks from the decompression buffer;

[0108] A block header parsing unit is used to parse the compressed block header to obtain the corresponding compressed block data length and compression type;

[0109] The data decompression unit is used to extract the corresponding compressed block data from the decompression buffer according to the length of the compressed block data, and decompress the compressed block data according to the compression type, so as to save the decompressed data to a preset original data buffer.

[0110] A data merging unit is used to merge the decompressed data in the original data buffer to obtain merged decompressed data, and to determine the length of the decompressed data of the merged decompressed data;

[0111] The data determination unit is used to compare the length of the decompressed data with the length of the original data. If the length of the decompressed data is the same as the length of the original data, the decompression is determined to be successful, and the merged decompressed data is determined to be the original data.

[0112] Furthermore, embodiments of this application also disclose an electronic device, Figure 5This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the ZSTD decompression method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0113] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0114] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0115] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the ZSTD decompression method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0116] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed ZSTD decompression method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0120] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A ZSTD decompression method, characterized in that, include: The compressed frame header is initialized, and the original data is divided into blocks to obtain several corresponding data blocks. The uncompressed data blocks in the several data blocks are ZSTD compressed to obtain corresponding compressed blocks. The compressed frame header includes a target field that characterizes the total length of the compressed data. Update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file; When decompressing the compressed frame, the compressed frame header of the compressed frame is parsed to determine the total length of the compressed data based on the obtained target field; The range of data to be read is determined based on the total length of the compressed data, and the corresponding target data is read from the disk file according to the range of data to be read, and the target data is decompressed by ZSTD to obtain the original data.

2. The ZSTD decompression method according to claim 1, characterized in that, Before initializing the compressed frame header, the following is also included: The target descriptor field in several frame header descriptors of the compressed frame header is determined; the target descriptor field includes a descriptor field representing currently unused bits and a descriptor field representing reserved bits. Change the flag bit of the target descriptor field to a new flag bit that represents the total length of the compressed data, so as to add the target field to the frame header structure of the compressed frame header, and adjust the ZSTD compression end source code and decompression end source code accordingly. The rules used for the newly added flag bits are consistent with the rules used for the flag bits that characterize the length of the original data before compression; the length of the target field is determined based on the newly added flag bits.

3. The ZSTD decompression method according to claim 1, characterized in that, The step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks to obtain corresponding compressed blocks includes: ZSTD compression is performed on the uncompressed data blocks in the plurality of data blocks to obtain the corresponding compression result; the compression result includes compressed block data; Based on the compression result, a corresponding compressed block header is generated to obtain a compressed block including the compressed block header and the compressed block data; The compressed block header is saved to a preset compressed block header buffer, and the compressed block data is saved to a preset compressed block data buffer.

4. The ZSTD decompression method according to claim 3, characterized in that, Updating the target field in the compressed frame header based on the compressed block includes: The compressed block header is obtained and parsed from the compressed block header buffer to determine the compressed block length, and the compressed block length is added to the target field in the compressed frame header to generate the target compressed frame header; Accordingly, obtaining the compressed frame including the compressed frame header includes: Data encapsulation is performed based on the target compressed frame header, the compressed block header in the compressed block header buffer, and the compressed block data in the compressed block data buffer to obtain the compressed frame corresponding to the compressed original data.

5. The ZSTD decompression method according to claim 1, characterized in that, The step of parsing the compressed frame header to determine the total length of the compressed data based on the obtained target field includes: Read the compressed frame header of the compressed frame and obtain the frame identifier in the compressed frame header. Verify the format validity of the compressed frame header based on the frame identifier. After the verification is successful, parse the frame header descriptor of the compressed frame header and determine the corresponding descriptor flag bit. Based on the descriptor flags, the unparsed structure of the compressed frame header is parsed to determine the total length of the compressed data according to the obtained target field, and to determine the original data length of the original data.

6. The ZSTD decompression method according to any one of claims 1 to 5, characterized in that, The step of determining the data reading range based on the total length of the compressed data includes: Determine the end position of the compressed frame header in the compressed frame, and determine the header length of the compressed frame header; The sum of the end position, the frame header length, and the total length of the compressed data is determined, and the end position is determined as the starting position of the data reading range, and the sum is determined as the ending position of the data reading range.

7. The ZSTD decompression method according to claim 5, characterized in that, The process of performing ZSTD decompression on the target data to obtain the original data includes: The target data is saved to a preset decompression buffer, and the compressed block headers of several compressed blocks are read sequentially from the decompression buffer; Parse the compressed block header to obtain the corresponding compressed block data length and compression type; According to the length of the compressed block data, the corresponding compressed block data is extracted from the decompression buffer, and the compressed block data is decompressed according to the compression type, so as to save the decompressed data to the preset original data buffer. The decompressed data in the original data buffer is merged to obtain merged decompressed data, and the length of the decompressed data of the merged decompressed data is determined. The length of the decompressed data is compared with the length of the original data. If the length of the decompressed data is the same as the length of the original data, the decompression is determined to be successful, and the merged decompressed data is identified as the original data.

8. A ZSTD decompression device, characterized in that, include: The data block compression module is used to initialize the compression frame header, divide the original data into blocks to obtain several corresponding data blocks, and perform ZSTD compression on the uncompressed data blocks in the several data blocks to obtain corresponding compressed blocks; the compression frame header includes a target field characterizing the total length of the compressed data. The compressed frame saving module is used to update the target field in the compressed frame header based on the compressed block, and jump to the step of performing ZSTD compression on the uncompressed data blocks in the plurality of data blocks until all the plurality of data blocks are compressed to obtain a compressed frame including the compressed frame header, and save the compressed frame to a disk file. The frame header parsing module is used to parse the compressed frame header of the compressed frame when decompressing the compressed frame, so as to determine the total length of the compressed data according to the obtained target field; The data decompression module is used to determine the data reading range based on the total length of the compressed data, read the corresponding target data from the disk file according to the data reading range, and perform ZSTD decompression on the target data to obtain the original data.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the ZSTD decompression method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the ZSTD decompression method as described in any one of claims 1 to 7.

Images