A fault-tolerant data segmentation compression method, recovery method, device and system
By using data segmentation parallel compression and RAID redundancy technology, the fault tolerance problem of data compression methods in error environments is solved, and an efficient and universal data transmission and storage solution is achieved.
Patent Information
- Application Number
- CN202310162977.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-02-24
AI Technical Summary
Existing data compression methods are susceptible to errors during transmission or storage, lack effective fault tolerance, and have poor versatility.
A data segmentation parallel compression method is adopted, which combines the RAID concept to construct redundant additional data blocks. Fault tolerance is provided through the RAID error correction mechanism. Before data segmentation, an appropriate compression algorithm and RAID level are selected, and a metadata structure is designed for data packaging.
It improves data compression speed and versatility, has fault tolerance, reduces the number of data transmission retransmissions and the risk of storage loss, and is suitable for data transmission and storage.
Smart Images

Figure CN116248129B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data compression, and more specifically, to a fault-tolerant data segmentation compression method, recovery method, device, and system. Background Technology
[0002] Currently, commonly used data compression can be divided into two main categories: lossless compression and lossy compression. Lossless compression utilizes the redundancy of data for compression, including Huffman coding, run-length coding, arithmetic coding, and LZ coding. Lossy compression reconstructs data differently from the original data and is mostly used in fields such as graphics, image, and digital speech compression.
[0003] Most data compression methods focus on researching specific algorithms to improve data compression ratios and speeds, aiming to provide a more general data compression and decompression solution. Other data compression methods analyze the characteristics of business data, develop specific algorithms, and improve compression efficiency in niche areas. To increase the data compression rate, some methods segment the data before performing parallel compression. A small number of methods add redundant information strongly correlated with the characteristics of specific business data to give the compressed data a certain degree of fault tolerance during recovery.
[0004] Most of the methods mentioned above focus on the data compression ratio. During the data compression process, no redundant information is added. When compressed data encounters errors during transmission or storage, the compressed data cannot be effectively recovered. Other methods with fault tolerance capabilities are often targeted at specific data types in specific fields and have poor versatility. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a fault-tolerant data segmentation compression method, recovery method, device, and system. It improves compression speed through parallel data segmentation compression; it introduces the concept of RAID (Redundant Array of Independent Disks) while segmenting data, compressing the segmented data blocks into redundant additional data blocks, and then synchronously transmitting or storing the packaged data. When compressed data is corrupted, fault tolerance is provided through the error correction mechanism of RAID; and the method can be quickly derived and ported to various applications, regardless of whether it is specifically designed for a particular domain or business data, thus possessing strong versatility.
[0006] The objective of this invention is achieved through the following approach: A fault-tolerant data segmentation compression method includes the following steps: The data is divided into blocks and then compressed in parallel; after the data is divided into blocks, redundant additional data blocks are constructed to achieve information redundancy before compression. The construction of redundant additional data blocks to achieve information redundancy is specifically based on the calculation of additional data blocks according to the RAID level.
[0007] Furthermore, before data segmentation, the steps include: strategy selection, which involves determining the strategy through online analysis or based on configuration, and determining the specific compression algorithm and data redundancy method; In the online analysis and strategy determination, when performing online analysis based on the specific characteristics of the compressed data, the data analysis module dynamically calculates the appropriate compression algorithm and RAID level, and then outputs the compression algorithm and RAID level as input for subsequent processing. In the process of determining the strategy based on configuration, the configuration data module is used to read and maintain the rules of compression algorithms or RAID levels corresponding to different types of data. During data compression, after determining the data type based on the original data using type matching, the specific compression algorithm and RAID level are obtained from the configuration data module for subsequent processing.
[0008] Furthermore, the data segmentation includes the following sub-steps: Let the original data size be M, the number of blocks be N, and the size of each data block be... If M is divisible by N, The possible values are as follows:
[0009] If M is not divisible by N The possible values are as follows:
[0010] When M is not divisible by N, the last block is padded with zeros to make its data length consistent with the block length. After data is segmented, additional blocks are calculated based on the RAID level to achieve information redundancy. Redundancy is only achieved with at least one additional block. If redundancy is achieved through replication, the segmented data is directly copied. If redundancy is achieved through parity checking, it is implemented using an XOR operation, as shown below:
[0011] In the above formula, , … P represents the data block, P represents the additional parity block, and xor represents the XOR operation.
[0012] Furthermore, the parallel compression includes the following sub-steps: During the initialization phase, a thread pool is created, and the block data is compressed in parallel using multiple threads; During data compression, for each data block that needs to be compressed, an idle thread is selected from the thread pool to complete the compression task and output the compressed data.
[0013] Furthermore, after parallel compression, a data combination step is included: The data packaging structure is designed to be divided into a metadata area and a compressed data area; The metadata area records key information of each compressed block, and the compressed data area stores the compressed data of each block and redundant additional blocks.
[0014] Furthermore, the metadata includes: compression algorithm, number of blocks, RAID level, metadata checksum, original length of each block, compressed length of each block, checksum of each compressed block, start and end offsets of each compressed block in the compressed data area, original length of redundant append blocks, compressed length of redundant append blocks, checksum of redundant append blocks, and start and end offsets of redundant append blocks in the compressed data area.
[0015] A fault-tolerant data segmentation recovery method includes the following steps: Obtain the data compressed using any of the compression methods described above, and recover the complete data based on the metadata and compressed data blocks in the data packet.
[0016] Furthermore, the step of recovering complete data based on metadata and compressed data blocks in the data packet includes the following sub-steps: S1, obtain the compression algorithm and RAID level; S2, determine whether the compressed block verification is successful. If so, decompress the data according to the compression algorithm and proceed to step S4; otherwise, proceed to the next step. S3: Determine if the additional compressed block verification was successful. If so, decompress the compressed block and additional block data according to the compression algorithm. Then, determine if it can be recovered according to the RAID level. If it can be recovered, recover the compressed block data that failed verification. If not, end the process.
[0017] S4, combine data according to the block configuration; S5, End.
[0018] A computer device includes a processor and a memory, the memory storing a computer program that, when loaded by the processor, executes the fault-tolerant data segmentation compression method as described above; or executes the fault-tolerant data segmentation recovery method as described above.
[0019] A computer system comprising the computer equipment described above.
[0020] The beneficial effects of this invention include: This invention implements a data compression method. First, a strategy is selected to determine the compression algorithm and RAID level. Then, data is segmented into blocks to achieve parallel compression and data redundancy. Finally, compressed data is packaged through metadata design. Compared to traditional methods, this method improves compression speed while possessing strong versatility, applicable to both data transmission and data storage. It also exhibits fault tolerance when compressed data is corrupted. For data transmission, it reduces the number of retransmissions; for data storage, it reduces the risk of data loss. Based on this, a compressed data recovery method, as well as related equipment and systems, are also implemented. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 Schematic diagram of fault-tolerant data segmentation compression principle; Figure 2 Schematic diagram of fault-tolerant data segmentation recovery principle; Figure 3 Here is a flowchart of the segmented data compression process; Figure 4 Choose a strategy diagram for online analytics; Figure 5 A diagram showing the strategy selection based on configuration; Figure 6 A schematic diagram showing how M can be divided by N; Figure 7 A schematic diagram showing the partitioning of M into blocks where M is not divisible by N; Figure 8 This is a timing diagram for parallel compression; Figure 9 This is a diagram illustrating the combination of data. Figure 10 Flowchart for segmented compressed data recovery; Figure 11 A diagram illustrating the system composition for an implementation example; Figure 12 This is a schematic diagram illustrating the principle of RAID5. Detailed Implementation
[0023] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.
[0024] This invention proposes a fault-tolerant, fast data segmentation compression method based on RAID principles. RAID, short for "disk array," connects multiple data storage areas in a logical manner, providing both data redundancy and improved read / write speeds. Based on performance, fault tolerance, and scalability, RAID can be divided into different levels, with typical RAID level characteristics shown in Table 1 below.
[0025] Table 1
[0026] This invention achieves parallel compression through data segmentation, thereby improving compression speed. After data segmentation, additional data blocks are calculated based on the RAID level to achieve information redundancy and provide fault tolerance in the event of data corruption. Its basic principle is as follows: Figure 1 As shown.
[0027] The basic principles of data recovery are as follows Figure 2 As shown below, the methods and processes for data compression and data recovery are described in detail.
[0028] I. Data Compression The processing flow for implementing the above method is as follows: Figure 3 As shown. According to Figure 1 and Figure 3 The data compression of this invention is divided into four parts: strategy selection, data segmentation, parallel compression, and data combination. Among them, the data combination part mainly describes the design of metadata.
[0029] (1) Strategy selection This invention does not specify which compression algorithm to use when implementing data compression, nor does it specify a specific RAID level for implementing data redundancy. The specific compression algorithm and data redundancy method can be determined through online analysis or based on configuration.
[0030] Online analysis determines the strategy: When performing online analysis based on the specific characteristics of the compressed data, a dedicated data analysis module dynamically calculates the appropriate compression algorithm and RAID level, such as... Figure 4 As shown. This invention does not involve specific data feature analysis algorithms, but its decision output compression algorithm and RAID level serve as inputs for subsequent processing.
[0031] Strategy determined by configuration: When implementing this invention, if there is sufficient prior knowledge and the appropriate compression algorithm and RAID level for different types of data are clearly defined, the compression algorithm and RAID level can be quickly determined through simple configuration. Figure 5 As shown. Figure 5In the configuration data module, the compression algorithm or RAID level rules corresponding to different types of data are read and maintained. During data compression, the type matching module determines the data type based on the original data and then obtains the specific compression algorithm and RAID level from the configuration data module for subsequent processing.
[0032] (2) Data partitioning To improve data compression speed and achieve data redundancy, the original data is divided into blocks. The number of blocks is determined based on the RAID level defined in 5.1.1 and the data volume, with the following basic principles: a) If the data volume is small, it may not need to be divided into blocks or may need to be divided into fewer blocks; b) If the data volume is large, it can be divided into multiple blocks; c) The selected RAID level implements data redundancy using a parity method, with a block count ≥ 2; d) After partitioning, the size of each data block should be as consistent as possible.
[0033] Suppose the original data size is M bytes, the number of blocks is N, and the size of each data block is... If M is divisible by N, such as Figure 6 As shown, The possible values are as follows:
[0034] If M is not divisible by N The possible values are as follows:
[0035] When M is not divisible by N, the last block needs to be padded with zeros to make its data length consistent with the lengths of the preceding blocks, such as... Figure 7 As shown.
[0036] After the data is divided into blocks, additional blocks are calculated according to the RAID level to achieve information redundancy. Redundancy can only be achieved if the number of additional blocks is greater than or equal to 1. If redundancy is achieved by replication, the block data can be copied directly. If redundancy is achieved by parity check, it is achieved by XOR operation, as shown below.
[0037]
[0038] In the above formula, , … Let P be the data block, P be the additional parity block, and XOR be the exclusive OR operation. The verification principle of XOR is shown in Table 2 below.
[0039] Table 2
[0040] In Table 2 above, values A and B each represent 1 bit values. When A and B are the same, the XOR result is 0; when A and B are different, the XOR result is 1. Knowing the XOR result and either A or B, the other value can be uniquely deduced. For example, if A = 1 and the XOR result is 1, then B = 0; if the XOR result is 0, then B = 1. This checksum method achieves information redundancy, allowing the original data to be recovered through XOR operations even when some information is corrupted.
[0041] (3) Parallel compression After data is segmented and additional data blocks are calculated based on the RAID level to achieve information redundancy, the segmented data is compressed in parallel using multiple threads to improve compression efficiency. To support data compression with different numbers of segments, a thread pool can be created for parallel processing. During initialization, a certain number of threads are created through the thread pool; during data compression, for each data segment that needs to be compressed, an idle thread is selected from the thread pool to complete the compression task, and the compressed data is output, with the timing as follows. Figure 8 As shown.
[0042] (4) Data combination After parallel data compression, the compressed blocks are of inconsistent length and need to be organized according to certain rules in order to recover the data during the decompression stage. The data packaging structure of this invention is as follows: Figure 9 As shown, it is divided into a metadata area and a compressed data area. The metadata area records key information such as the length of each compressed block, the original length, the checksum, and the compression algorithm. The compressed data area stores the compressed data of each block and redundant additional blocks.
[0043] The metadata design should include the following: a) Compression algorithm; b) Number of blocks; c) RAID levels; d) Metadata verification value; e) Original length of each data block; f) The length of each compressed data block; g) Verification values of each compressed data block; h) The start and end offsets of each data block in the compressed data area after compression; i) The original length of the redundant append block; j) The compressed length of the redundant additional block; k) Verification value of redundant additional blocks after compression; I) The start and end offsets of the redundant additional block in the compressed data area after compression.
[0044] II. Data Recovery The data compressed using the above method is then used to reconstruct the complete data based on the metadata and compressed data blocks in the data packet. The process is as follows: Figure 10 As shown.
[0045] In this invention example, the system composition is as follows: Figure 11 As shown, during the data interaction between terminal A and terminal B, the data volume is large and the data link bandwidth is low. The redundancy mechanism of this invention achieves data fault tolerance, reduces the number of data retransmissions, and improves transmission efficiency. In the example, when the application software sends data, it first undergoes data segmentation and compression by the data segmentation and compression module to reduce the data volume, and then the data is sent through the data link by the data transceiver module. After the data transceiver module receives the data, it recovers the compressed data through the data segmentation and compression module before sending it to the application software for processing.
[0046] Since the data types transmitted in this embodiment are few and the data characteristics are clear, the compression strategy is determined by configuration when selecting the compression strategy, and RAID5 is used to implement data redundancy through parity checking. Figure 12 As shown.
[0047] The data partitioning strategies for each type of data are shown in Table 3 below.
[0048] Table 3
[0049] During data compression, the number of blocks is as shown in Table 3 above. After determining the number of blocks, proceed according to... Figure 6 , Figure 7 Determine the data size of each data block.
[0050] In the implementation of the segmented data compression module, two thread pools are created for parallel data compression and parallel data recovery, respectively, with each thread pool containing 12 threads.
[0051] The metadata design is shown in Table 4 below. The table uses CRC32 for data verification. If data compression is not performed, the original data description information is stored in the relevant fields of block data 1.
[0052] Table 4
[0053] After receiving the data, the data transceiver module recovers the data based on the metadata description information, such as... Figure 10 As shown, in order to improve the decompression efficiency of compressed data, a thread pool is used for parallel recovery, and the original data is reassembled after each compressed data block is decompressed.
[0054] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.
[0055] Example 1 A fault-tolerant data segmentation compression method includes the following steps: The data is divided into blocks and then compressed in parallel; after the data is divided into blocks, redundant additional data blocks are constructed to achieve information redundancy before compression. The construction of redundant additional data blocks to achieve information redundancy is specifically based on the calculation of additional data blocks according to the RAID level.
[0056] Example 2 Based on Example 1, before data segmentation, the following steps are also included: strategy selection, determining the strategy through online analysis or based on configuration, and determining the specific compression algorithm and data redundancy method; In the online analysis and strategy determination, when performing online analysis based on the specific characteristics of the compressed data, the data analysis module dynamically calculates the appropriate compression algorithm and RAID level, and then outputs the compression algorithm and RAID level as input for subsequent processing. In the process of determining the strategy based on configuration, the configuration data module is used to read and maintain the rules of compression algorithms or RAID levels corresponding to different types of data. During data compression, after determining the data type based on the original data using type matching, the specific compression algorithm and RAID level are obtained from the configuration data module for subsequent processing.
[0057] Example 3 Based on Example 1, the data segmentation includes the following sub-steps: Let the original data size be M, the number of blocks be N, and the size of each data block be... If M is divisible by N, The possible values are as follows:
[0058] If M is not divisible by N The possible values are as follows:
[0059] When M is not divisible by N, the last block is padded with zeros to make its data length consistent with the block length. After data is segmented, additional blocks are calculated based on the RAID level to achieve information redundancy. Redundancy is only achieved with at least one additional block. If redundancy is achieved through replication, the segmented data is directly copied. If redundancy is achieved through parity checking, it is implemented using an XOR operation, as shown below:
[0060] In the above formula, , … P represents the data block, P represents the additional parity block, and xor represents the XOR operation.
[0061] Example 4 Based on Example 1, the parallel compression includes the following sub-steps: During the initialization phase, a thread pool is created, and the block data is compressed in parallel using multiple threads; During data compression, for each data block that needs to be compressed, an idle thread is selected from the thread pool to complete the compression task and output the compressed data.
[0062] Example 5 Based on Example 1, after parallel compression, the data combination step is included: The data packaging structure is designed to be divided into a metadata area and a compressed data area; The metadata area records key information of each compressed block, and the compressed data area stores the compressed data of each block and redundant additional blocks.
[0063] Example 6 Based on Example 5, the metadata includes: compression algorithm, number of blocks, RAID level, metadata check value, original length of each block, compressed length of each block, check value of each block, start and end offsets of each compressed block in the compressed data area, original length of redundant supplementary blocks, compressed length of redundant supplementary blocks, check value of redundant supplementary blocks, and start and end offsets of redundant supplementary blocks in the compressed data area.
[0064] Example 7 A fault-tolerant data segmentation recovery method includes the following steps: Obtain the data compressed by any of the compression methods described in Examples 1 to 6, and recover the complete data based on the metadata and compressed data blocks in the data packet.
[0065] Example 8 Based on Example 7, the step of recovering complete data from metadata and compressed data blocks in the data packet includes the following sub-steps: S1, obtain the compression algorithm and RAID level; S2, determine whether the compressed block verification is successful. If so, decompress the data according to the compression algorithm and proceed to step S4; otherwise, proceed to the next step. S3: Determine if the additional compressed block verification was successful. If so, decompress the compressed block and additional block data according to the compression algorithm. Then, determine if it can be recovered according to the RAID level. If it can be recovered, recover the compressed block data that failed verification. If not, end the process.
[0066] S4, combine data according to the block configuration; S5, End.
[0067] Example 9 A computer device includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes a fault-tolerant data segmentation compression method as described in any one of Embodiments 1 to 6; or executes a fault-tolerant data segmentation recovery method as described in Embodiment 7.
[0068] Example 10 A computer system includes the computer device as described in Example 9.
[0069] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0070] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.
[0071] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.
[0072] All parts not covered in this invention are the same as or can be implemented using existing technologies.
[0073] The above technical solution is only one embodiment of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the above specific embodiments of the present invention. Therefore, the methods described above are only preferred and are not restrictive.
[0074] In addition to the examples above, other embodiments may be obtained by those skilled in the art based on the above disclosure or by making modifications using knowledge or technology in related fields. The features of each embodiment may be interchanged or replaced. Modifications and changes made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.
Claims
1. A fault-tolerant data segmentation compression method, characterized in that, Includes the following steps: The data is divided into blocks and then compressed in parallel; after the data is divided into blocks, redundant additional data blocks are constructed to achieve information redundancy before compression. The construction of redundant additional data blocks to achieve information redundancy is specifically based on the calculation of additional data blocks according to the RAID level to achieve information redundancy; The data segmentation includes the following sub-steps: Let the original data size be M, the number of blocks be N, and the size of each data block be... If M is divisible by N, The possible values are as follows: If M is not divisible by N The possible values are as follows: When M is not divisible by N, the last block is padded with zeros to make its data length consistent with the block length. After data is segmented, additional blocks are calculated based on the RAID level to achieve information redundancy. Redundancy is only achieved with at least one additional block. If redundancy is achieved through replication, the segmented data is directly copied. If redundancy is achieved through parity checking, it is implemented using an XOR operation, as shown below: In the above formula, , … P represents the data block, P represents the additional parity block, and xor represents the XOR operation.
2. The fault-tolerant data segmentation compression method according to claim 1, characterized in that, Before data segmentation, the following steps are also included: strategy selection, which involves determining the strategy through online analysis or based on configuration, and determining the specific compression algorithm and data redundancy method. In the online analysis to determine the strategy, when performing online analysis based on the specific characteristics of the compressed data, the data analysis module dynamically calculates the appropriate compression algorithm and RAID level, and then outputs the compression algorithm and RAID level as input for subsequent processing. In the strategy determination process based on configuration, the configuration data module is used to read and maintain the compression algorithm or RAID level rules corresponding to different types of data. During data compression, type matching is used to determine the data type based on the original data, and then the specific compression algorithm and RAID level are obtained from the configuration data module for subsequent processing.
3. The fault-tolerant data segmentation compression method according to claim 1, characterized in that, The parallel compression includes the following sub-steps: During the initialization phase, a thread pool is created, and the block data is compressed in parallel using multiple threads; During data compression, for each data block that needs to be compressed, an idle thread is selected from the thread pool to complete the compression task and output the compressed data.
4. The fault-tolerant data segmentation compression method according to claim 1, characterized in that, Following parallel compression, the data combination step is included: The data packaging structure is designed to be divided into a metadata area and a compressed data area; The metadata area records key information of each compressed block, and the compressed data area stores the compressed data of each block and redundant additional blocks.
5. The fault-tolerant data segmentation compression method according to claim 4, characterized in that, The metadata includes: compression algorithm, number of blocks, RAID level, metadata checksum, original length of each block, compressed length of each block, checksum of each compressed block, start and end offsets of each compressed block in the compressed data area, original length of redundant append blocks, compressed length of redundant append blocks, checksum of redundant append blocks, and start and end offsets of redundant append blocks in the compressed data area.
6. A fault-tolerant data segmentation recovery method, characterized in that, Includes the following steps: Obtain the data compressed by any one of the compression methods described in claims 1 to 5, and recover the complete data based on the metadata and compressed data blocks in the data packet.
7. The fault-tolerant data segmentation recovery method according to claim 6, characterized in that, The process of recovering complete data based on metadata and compressed data blocks in the data packet includes the following sub-steps: S1, obtain the compression algorithm and RAID level; S2, determine whether the compressed block verification is successful. If so, decompress the data according to the compression algorithm and proceed to step S4; otherwise, proceed to the next step. S3, determine whether the additional compressed block verification is successful. If so, decompress the compressed block and additional block data according to the compression algorithm. Next, determine whether recovery is possible based on the RAID level. If it is possible, recover the compressed block data that failed to be verified; otherwise, end the process. S4, combine data according to the block configuration; S5, End.
8. A computer device, characterized in that, The computer device includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the fault-tolerant data segmentation compression method as described in any one of claims 1 to 5. Alternatively, the fault-tolerant data segmentation recovery method as described in claim 6 may be implemented.
9. A computer system, characterized in that, Includes the computer device as described in claim 8.
Citation Information
Patent Citations
Hardware acceleration of commonality factoring with removable media
US20090013140A1
