Data compression processing method and system

Abstract

The invention provides a data compression processing method and system, and the method comprises the steps: obtaining a backup data stream corresponding to a current data compression processing task, and carrying out the data segmentation processing of the backup data stream, and obtaining a plurality of data segments; writing the plurality of data segments into a data backup system, and if it is determined that data blocks in the data segments are sequentially distributed in a container in the data backup system according to the container reference rate of the container in the data backup system, performing data deduplication processing on the first data block, the first data block is a data block in which the data blocks in the data segment are sequentially distributed in a container in the data backup system; and local joint deduplication processing is carried out on other data blocks except the first data block in the container in the data backup system to complete the current data compression processing task, and the local joint deduplication processing comprises multi-container data deduplication processing and differential compression processing. When data compression processing is carried out, the performance and the reliability of the data backup system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems, and in particular to a data compression processing method and system. Background Art

[0002] In order to cope with the increasingly serious data redundancy problem in current backup storage, data deduplication and differential compression technologies have emerged. Data deduplication technology significantly reduces the amount of data that needs to be physically stored by identifying and eliminating duplicate data blocks in the data set. Differential compression focuses on the changed parts of the existing backup data and only stores the new or modified data blocks compared to the previous backup, thereby achieving more efficient storage utilization.

[0003] Although differential compression technology has shown great potential in improving backup storage efficiency, it also brings a series of challenges to data backup systems. First, the process of searching and matching similar data consumes a lot of computing resources, especially when dealing with large-scale data sets. This overhead may become considerable, affecting the performance of backup and recovery operations. Second, for backup systems with high real-time requirements, complex data deduplication and differential compression processing may increase the time delay of data recovery. Especially in emergency situations, this delay may directly lead to data unavailability, exacerbating the risk of data loss and economic loss.

[0004] Therefore, there is an urgent need for a data compression processing method and system to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a data compression processing method and system.

[0006] The present invention provides a data compression processing method, comprising: Acquire a backup data stream corresponding to the current data compression processing task, and perform data segmentation processing on the backup data stream to obtain multiple data segments; Writing the multiple data segments into a data backup system, and if it is determined, based on a container reference rate of a container in the data backup system, that data blocks in the data segment are sequentially distributed in a container in the data backup system, performing data deduplication processing on a first data block, wherein the first data block is a data block in which data blocks in the data segment are sequentially distributed in a container in the data backup system; Local joint redundancy elimination processing is performed on other data blocks in the container of the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0007] According to a data compression processing method provided by the present invention, after performing local joint redundancy elimination processing on other data blocks except the first data block in the container in the data backup system to complete the current data compression processing task, the method further includes: Reading the data blocks in the corresponding container in the data backup system according to the sequence of the data blocks in the file spectrum corresponding to the current data compression processing task; If the read data block is a data block after differential compression processing, obtain the corresponding reference data block, and perform differential decoding processing on the data block after differential compression processing according to the reference data block to obtain a data block after data recovery.

[0008] According to a data compression processing method provided by the present invention, the method further includes: In the process of writing the plurality of data segments into the data backup system, determining a corresponding container number in the data backup system according to a data block type of each data block in the plurality of data segments, wherein the data block type includes a duplicate data block, a differential block, a reference block, and a non-duplicate and non-differentially compressed data block; According to the container number, a recovery cache state during data recovery is simulated.

[0009] According to a data compression processing method provided by the present invention, if the read data block is a data block after differential compression processing, a corresponding reference data block is obtained, and according to the reference data block, a differential decoding process is performed on the data block after differential compression processing to obtain a data block after data recovery, the method further includes: Calculate the recovery factor according to the data size of the data block after the data is recovered and the corresponding container read quantity; According to the mapping relationship between the recovery factor and the redundancy elimination strategy, a target redundancy elimination strategy corresponding to the next data compression processing task is obtained.

[0010] According to a data compression processing method provided by the present invention, if it is determined according to the container reference rate of the container in the data backup system that the data blocks in the data segment are sequentially distributed in the container in the data backup system, performing data deduplication processing on the first data block includes: Obtaining the total size of data blocks in the data segment written into the container; Calculating the container reference rate corresponding to the container according to the total size of the data blocks and the total storage space of the container; According to the container reference rate and a preset reference rate threshold, determining whether all data blocks in the container are data blocks in the data segment; If all the data blocks in the container are data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is sequential distribution; Perform data deduplication processing on sequentially distributed data blocks; If the data blocks in the container are part of the data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is non-sequential distribution; Write the non-sequentially distributed data blocks to the corresponding container.

[0011] According to a data compression processing method provided by the present invention, the local joint redundancy elimination processing is performed on other data blocks in the container in the data backup system except the first data block, including: Performing data deduplication processing on the second data blocks in each container in the data backup system, wherein the second data blocks are data blocks in the data segment other than the first data blocks and determined to be duplicate data; According to the characteristic value of the third data block, a corresponding reference data block is matched from the data backup system, and if the third data block is determined to be a similar data block based on the reference data block, differential compression processing is performed on the third data block; The third data block is a data block in the data segment except the first data block and the second data block.

[0012] According to a data compression processing method provided by the present invention, the corresponding reference data block is matched from the data backup system according to the characteristic value of the third data block, and if the third data block is determined to be a similar data block based on the reference data block, differential compression processing is performed on the third data block, including: Based on a rolling hash algorithm, calculating multiple hash values ​​corresponding to the third data block; Based on a preset feature value extraction algorithm, determining a feature value of the third data block from the multiple hash values; According to the characteristic value of the third data block, a plurality of target historical data blocks are matched from the data backup system, wherein the target historical data block is a historical data block in a historical characteristic value table of the data backup system having the same characteristic value as the characteristic value of the third data block, and the historical characteristic value table is constructed by characteristic values ​​corresponding to the historical data blocks; Determine the target historical data block with the largest number of matches with the feature value of the third data block among the multiple target historical data blocks as the reference data block; Determine whether there is a difference between the reference data block and the third data block, and if there is a difference, determine that the third data block is a similar data block; The third data block is subjected to differential compression processing.

[0013] The present invention also provides a data compression processing system, comprising: A data segmentation module is used to obtain a backup data stream corresponding to the current data compression processing task, and perform data segmentation processing on the backup data stream to obtain multiple data segments; a data deduplication object selection module, configured to write the plurality of data segments into a data backup system, and if it is determined, based on a container reference rate of a container in the data backup system, that data blocks in the data segment are sequentially distributed in a container in the data backup system, perform data deduplication processing on a first data block, wherein the first data block is a data block in which data blocks in the data segment are sequentially distributed in a container in the data backup system; A local joint redundancy elimination module is used to perform local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned data compression processing methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data compression processing method described above is implemented.

[0016] The data compression processing method and system provided by the present invention perform data segmentation processing on the backup data stream corresponding to the current data compression processing task, and then write multiple data segments into the data backup system. If it is determined that the data blocks in the data segment are sequentially distributed in the container based on the container reference rate, the data blocks in the sequential distribution in the container are deduplicated, and then other data blocks in the container in the data backup system are locally jointly deduplicated to complete the current data compression processing task, thereby improving the performance and reliability of the data backup system when performing data compression processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a data compression processing method provided by the present invention; Figure 2 A structural schematic diagram of a data compression processing system provided by the present invention; Figure 3 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In the backup storage system, there are a large number of non-duplicate data blocks, which still contain a lot of identical data. Although data deduplication technology can reduce some redundant data, it cannot completely eliminate all redundancy.

[0021] In order to further save storage space, differential compression can be used in related technologies to eliminate redundant data in non-duplicate data blocks. However, the search and processing of similar data will have a negative impact on the recovery performance of the backup system, especially for backup systems that rely heavily on real-time data. In an emergency, fast data recovery can minimize data loss and economic losses, so the recovery speed in the backup storage system becomes crucial.

[0022] In the related art, after differential compression is applied to a backup system based on deduplication, differential compression needs to read the contents of the reference block and the target block to perform differential encoding and decoding operations. Compared with directly reading the target block, differential compression requires more disk IO operations, thereby increasing the number of disk reads, causing the data backup system to face problems such as disk bottlenecks and reduced throughput, which in turn affects the performance of the backup system.

[0023] Therefore, when improving the performance and reliability of the data backup system, it is necessary to balance the storage space and recovery speed and select the appropriate data compression technology and recovery solution. For industries that rely heavily on real-time data, the performance and reliability of the backup system are particularly critical.

[0024] Figure 1 A flow chart of the data compression processing method provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a data compression processing method, comprising: Step 101, obtaining a backup data stream corresponding to a current data compression processing task, and performing data segmentation processing on the backup data stream to obtain a plurality of data segments.

[0025] In the present invention, the currently ongoing data compression processing task is first identified and locked. Each data compression processing task is associated with a specific data set. These data sets come from multiple channels such as databases, file systems, and network transmissions. Through these channels, the corresponding backup data streams are obtained in preparation for compression to reduce storage space occupancy.

[0026] After the backup data stream is obtained, it is segmented. Data segmentation is a process of dividing a data set into smaller, more manageable parts (i.e., data segments). In the present invention, it can be customized according to the characteristics of the data and the needs of the system. For example, segmentation can be performed according to a fixed size (such as 1MB, 10MB per segment, etc.), or intelligent segmentation can be performed based on data content (such as file boundaries, record separators, etc.). For backup data streams, since they may contain multiple types of data (such as text, pictures, videos, etc.), and the amount of data may be very large, a combination of one or more segmentation strategies is usually adopted to ensure the effectiveness and efficiency of segmentation.

[0027] After data segmentation, the original backup data stream is divided into multiple smaller data segments. These data segments are logically independent of each other, but they still maintain a physical association with the original data stream (for example, identified by index, sequence number or timestamp, etc.). Each data segment contains a part of the data in the original data stream and can be processed separately (such as compression, transmission, etc.).

[0028] Step 102, write the multiple data segments into the data backup system. If it is determined that the data blocks in the data segments are sequentially distributed in the container in the data backup system based on the container reference rate of the container in the data backup system, deduplicate the first data block, wherein the first data block is a data block in the data segment that is sequentially distributed in the container in the data backup system.

[0029] In the present invention, the backup data stream is divided into multiple data segments, and a queue is set for each data segment. When writing the backup data stream, it is first detected whether the data storage on the hard disk is distributed in sequence. When all the data blocks in the container belong to the data blocks of the currently cached data segment, it means that the data blocks in the container are stored in sequence, and if a container only contains a small number of duplicate data blocks in the data segment, it means that the data blocks in the container are distributed non-sequentially for the data segment.

[0030] Specifically, a data backup system is a system dedicated to storing, managing and protecting data. The process of writing data segments into the data backup system is actually to allocate these data segments to different physical or logical containers (such as files, partitions, volumes, etc.) for storage. In the data backup system, the container reference rate of each container reflects the frequency of data blocks in the container being referenced or accessed, thereby determining the distribution of data blocks in the container. Furthermore, by analyzing the source, size, type, and location of data blocks in the container, combined with the container reference rate, it is possible to determine whether the data blocks in the data segment are distributed sequentially in the container.

[0031] In the present invention, sequentially distributed data blocks indicate that the data blocks are continuous or nearly continuous in physical storage, which is beneficial for improving read and write efficiency, reducing fragmentation, and optimizing backup and recovery processes. When the container utilization rate reaches 100%, all data blocks in the container belong to the currently processed data segment, thereby determining that the data blocks are sequentially distributed in the container.

[0032] Furthermore, after it is determined that data blocks in a certain data segment are sequentially distributed in a container in the data backup system, data deduplication is performed on these sequentially distributed data blocks in the data segment. Data deduplication is a technology for reducing data redundancy and optimizing storage space, which is achieved by identifying and deleting duplicate data blocks.

[0033] During the data deduplication process, it is checked whether each data block is the same as the previous stored data block. If so, the data block will not be actually written to the storage medium, but only a reference or pointer to the stored data block will be recorded, so that the amount of data finally stored in the data backup system will be greatly reduced. It should be noted that the first data block in the present invention actually represents any one or more of all data blocks in the data segment that are determined to be distributed sequentially in the container, and the data deduplication process will be performed one by one or in batches for these sequentially distributed data blocks.

[0034] Step 103, performing local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0035] In the present invention, after completing the data deduplication processing for sequentially distributed data blocks, it is necessary to further optimize the remaining data blocks in the container in the data backup system (i.e., other data blocks except the first data block) to eliminate redundant data therein, i.e., local joint redundancy elimination processing. By combining multiple technical means, data redundancy is reduced in a wider range (including multiple containers) and storage efficiency is improved.

[0036] Specifically, the first step of the local joint redundancy elimination process is multi-container data deduplication processing, which searches and deletes duplicate data blocks in multiple containers. Different from the deduplication for a single data segment or container in the above embodiments, multi-container data deduplication expands the field of view to multiple containers in the entire data backup system, thereby comparing data blocks across containers, identifying those data blocks that are actually the same in content although distributed in different containers, and deleting them as redundant data.

[0037] In the present invention, the multi-container data deduplication processing process includes data blocks in all containers written since sequential deduplication, which can more effectively discover and utilize similarities between data and reduce the amount of redundant data in the entire data backup system.

[0038] Furthermore, after the multi-container data deduplication process is completed, differential compression is performed. Differential compression is a compression technology that optimizes data blocks that are similar but not completely duplicated. In a data backup system, there are often a large number of data blocks with similar but not completely identical content, which may be generated due to minor modifications or updates. Differential compression technology reduces the use of storage space by storing only the difference between data blocks (i.e., the difference) instead of the entire data block itself.

[0039] Specifically, in the present invention, the differential compression process in the local joint redundancy elimination process focuses on similar but non-duplicate data blocks that have not been deduplicated. For these data blocks, the similarities between them are first identified, and then the difference information (i.e., the difference) between them is calculated and stored. When it is necessary to restore these data blocks, the complete data blocks will be reconstructed based on the original data blocks and the differential information.

[0040] The present invention further optimizes the data blocks other than the first data block in the container of the data backup system through the local joint redundancy elimination processing of the above-mentioned multi-container data deduplication processing and differential compression processing. This processing process not only reduces data redundancy, but also improves the utilization rate of storage space and the efficiency of backup and recovery. Finally, the entire data compression processing task is completed, and the data backup system stores all necessary data in a more compact and efficient state.

[0041] The data compression processing method provided by the present invention performs data segmentation processing on the backup data stream corresponding to the current data compression processing task, and then writes multiple data segments into a data backup system. If it is determined that the data blocks in the data segment are sequentially distributed in the container based on the container reference rate, the data blocks in the sequential distribution in the container are deduplicated, and then other data blocks in the container in the data backup system are locally jointly deduplicated to complete the current data compression processing task, thereby improving the performance and reliability of the data backup system when performing data compression processing.

[0042] On the basis of the above embodiment, after performing local joint redundancy elimination processing on other data blocks except the first data block in the container in the data backup system to complete the current data compression processing task, the method further includes: Reading the data blocks in the corresponding container in the data backup system according to the sequence of the data blocks in the file spectrum corresponding to the current data compression processing task; If the read data block is a data block after differential compression processing, obtain the corresponding reference data block, and perform differential decoding processing on the data block after differential compression processing according to the reference data block to obtain a data block after data recovery.

[0043] In the present invention, the file spectrum corresponding to the current data compression processing task is first read, and the containers where the data blocks to be restored are located are read one by one according to the sequence of the data blocks in the file spectrum. Optionally, since the basic reading unit of the data backup system is a container, this process can reduce the number of disk reads through a recovery cache. In one embodiment, if the read data is a compressed differential block, its referenced data block is further read, and the original data block is restored by differential decoding. When a data segment of a certain length of a file is restored, the data segment is sent to the user end. When it is determined that all data segments of a file have arrived at the user end, the restored file is obtained by merging for use by the user end.

[0044] Specifically, in the present invention, the order of the data blocks to be restored is determined according to the file spectrum corresponding to the current data compression processing task (which contains the layout and sequence information of the data blocks), so as to ensure that the data can be correctly reorganized according to the logical structure of the original file during the recovery process. Then, according to the identification information of the data blocks in the file spectrum, the containers where the data blocks are located are located and read one by one. In order to improve the reading efficiency, during the reading process, the data blocks that have been read (or part of the contents of the container) will be cached, so that they can be directly obtained from the cache when needed later, without having to read from the disk again, thereby reducing the number of disk I / O operations and improving the overall recovery speed.

[0045] Furthermore, during the reading process, if it is identified that a data block has been subjected to differential compression processing (i.e., the data block is compressed based on the difference of one or more existing reference data blocks), the differential decoding process will be triggered. In order to perform differential decoding, the present invention first obtains the reference data block on which the differential compressed data block depends. The process is to find and read the corresponding reference data block according to the reference information in the differential compressed data block (such as the identifier or position information of the reference data block). Once the reference data block is obtained, the differential compressed data block can be processed using a differential decoding algorithm. The differential decoding algorithm uses the difference information in the reference data block and the differential compressed data block to reconstruct the original data block.

[0046] As data blocks are recovered from the original data through differential decoding, they are integrated into the file being recovered. As the individual data segments of the file are gradually recovered, when all the data segments of the file are successfully recovered and transmitted to the user end, the data segments are merged into a complete file.

[0047] Based on the above embodiment, the method further includes: In the process of writing the plurality of data segments into the data backup system, determining a corresponding container number in the data backup system according to a data block type of each data block in the plurality of data segments, wherein the data block type includes a duplicate data block, a differential block, a reference block, and a non-duplicate and non-differentially compressed data block; According to the container number, a recovery cache state during data recovery is simulated.

[0048] The data in the real recovery cache is a container containing a large number of data blocks. In order to save memory, the present invention uses container numbers to simulate the state of the data cache. At the beginning of the backup, a recovery cache is first initialized. When a data block is processed by redundancy elimination, its container ID is used to update the recovery cache. If the data block has been differentially compressed or is a copy of a previously written differential, the container ID of the referenced data block is also used to update the recovery cache.

[0049] When the data backup is completed, different types of data blocks can be identified by calculating the speed factor. The speed factor can be calculated according to the recovery efficiency in the data recovery process, so as to evaluate the data recovery performance. In the present invention, the container number is an 8-byte integer, and different types of data blocks can find the corresponding container in the storage system through the value corresponding to the container number. During data backup, data blocks can be divided into: duplicate data blocks, differential blocks, reference blocks of differential blocks, and ordinary data blocks that are not duplicated and have not been differentially compressed. The recovery cache can be simulated by using the container numbers of these four types of data blocks. When the data blocks are written to the hard disk, the containers where the four types of data blocks are located have been obtained, and the container numbers of the above four types of data blocks are used to update the recovery cache. For duplicate data blocks, their container numbers can be obtained when redundancy is eliminated; for differential blocks and ordinary data blocks, since they will be written into new containers, the container numbers can be obtained when they are written to the hard disk; for reference blocks, their container numbers can be obtained in the similar data retrieval link.

[0050] In the present invention, at the beginning of the backup, the simulated recovery cache is initialized according to the predetermined recovery cache size. This cache does not store actual data, but stores a mapping or list of container numbers. When new data blocks are written and allocated to the container, the container numbers of these data blocks are used to update the simulated recovery cache. For the differential block, the container number of its referenced block also needs to be recorded.

[0051] After the backup is completed, the simulated recovery cache can be used to simulate the recovery process, evaluate the number, order and mode of containers that may need to be read during recovery, and help optimize the actual recovery strategy, for example, by adjusting the data layout to improve the recovery speed. The present invention can evaluate and optimize the performance of data backup and recovery without actually performing the recovery operation by recording the container numbers of various data blocks and simulating the state of the recovery cache.

[0052] On the basis of the above embodiment, if the read data block is a data block after differential compression processing, the corresponding reference data block is obtained, and according to the reference data block, the differential compression data block is differentially decoded to obtain the data block after data recovery, the method further includes: Calculate the recovery factor according to the data size of the data block after the data is recovered and the corresponding container read quantity; According to the mapping relationship between the recovery factor and the redundancy elimination strategy, a target redundancy elimination strategy corresponding to the next data compression processing task is obtained.

[0053] In the present invention, the redundancy elimination strategy to be adopted this time can be determined by evaluating the recovery performance of the last backup. To achieve this, a recovery cache is simulated during data backup, and the recovery factor of this backup is calculated after the backup is completed, and written into a file, so that the file will be read during the next backup, and the redundancy elimination strategy to be adopted is determined based on the recovery performance factor. It should be noted that in the present invention, the recovery factor represents the reciprocal of the number of containers required to be read for each 1MB of data to be restored, that is, the calculation formula of the recovery factor is the data volume / the number of containers read during the recovery, and the larger the recovery factor, the faster the recovery speed.

[0054] Specifically, in the present invention, in order to evaluate the recovery performance of the backup during the data backup process, a recovery cache is simulated to simulate the data recovery process. This process records the number of containers required to read each data block during recovery. When the backup is completed, the recovery factor is calculated based on the data collected during the backup process.

[0055] After the recovery factor is calculated, the redundancy elimination strategy to be adopted for the next data compression processing task is determined according to the mapping relationship between the recovery factor and the redundancy elimination strategy. In the present invention, different redundancy elimination strategies are specified according to different ranges of the recovery factor. For example, if the recovery factor is very high (indicating good recovery performance), the system may select a relatively simple but low compression redundancy elimination strategy to save processing time and resources; and if the recovery factor is low (indicating poor recovery performance), the system may select a more complex but higher compression strategy to save more space on the backup storage, but may sacrifice some recovery speed.

[0056] In the actual data backup process, a recovery process is simulated to record the number of containers that need to be read when restoring each data block. After the backup is completed, the recovery factor is calculated based on the total amount of data backed up and the number of containers read during the recovery, and the calculated recovery factor is written to a file or database for subsequent use. Before the next backup or data compression task starts, the recovery factor file is read, and the target redundancy elimination strategy to be adopted is determined based on the mapping relationship between the recovery factor and the redundancy elimination strategy, so as to dynamically adjust the redundancy elimination strategy according to the actual recovery performance to achieve the best balance between backup storage efficiency and recovery speed.

[0057] On the basis of the above embodiment, if it is determined according to the container reference rate of the container in the data backup system that the data blocks in the data segment are sequentially distributed in the container in the data backup system, performing data deduplication processing on the first data block includes: Obtaining the total size of data blocks in the data segment written into the container; Calculating the container reference rate corresponding to the container according to the total size of the data blocks and the total storage space of the container; According to the container reference rate and a preset reference rate threshold, determining whether all data blocks in the container are data blocks in the data segment; If all the data blocks in the container are data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is sequential distribution; Perform data deduplication processing on sequentially distributed data blocks; If the data blocks in the container are part of the data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is non-sequential distribution; Write the non-sequentially distributed data blocks to the corresponding container.

[0058] When performing sequential deduplication, if the speed factor of the current backup may be only slightly higher, if the redundancy elimination operation is continued, the data blocks in the storage will be referenced, resulting in further reduction in recovery performance. By skipping the redundancy elimination step and directly writing the data to the storage system, a loss of compression ratio may occur. In the present invention, by determining whether the data blocks are distributed sequentially, these data blocks are identified so that the current backup only references them. When taking the sequential deduplication step, the data stream is first divided into multiple data segments, and a queue is set for each data segment. When writing the backup data stream, it is detected whether the data is stored in a sequential distribution on the hard disk. When all the data blocks in the container belong to the data blocks of the cached data segment, the data blocks in the container are stored sequentially. If a container only contains a small number of duplicate data blocks in the data segment, the data blocks in the container are non-sequentially distributed for the data segment.

[0059] When performing sequential deduplication, it is first necessary to determine which data blocks in which containers can be deduplicated. The present invention counts the data blocks in a single data segment and calculates their reference rates. This process needs to consider all containers referenced by the data blocks and calculate their container reference rates. In the present invention, when calculating the container reference rate, the numerator in the formula includes the total size of data blocks of multiple data block types, such as the total size of duplicate data blocks, differential blocks, and referenced data blocks. When the calculated container utilization rate reaches 100% (i.e., the preset reference rate threshold), it is considered that the data blocks in the container are distributed sequentially for the data segment, indicating that the data blocks that are duplicated with the data blocks in the container will be considered as duplicate data blocks; otherwise, the data blocks in the data segment will be written into the container again.

[0060] Specifically, in the present invention, the total size of all data blocks in a specific data segment written into a container is obtained, and at the same time, the total storage space size of the container is obtained. Then, the container reference rate is calculated based on the total size of the data blocks and the total storage space of the container. According to the calculated container reference rate and a preset reference rate threshold (this threshold is usually set according to system performance requirements and experience), it can be determined whether all the data blocks in the container belong to the data segment. If the container reference rate is higher than or equal to the preset threshold, and all the data blocks in the container belong to the data segment, it is considered that the data blocks are distributed sequentially in the container; otherwise, when the container reference rate is lower than the threshold, the container only contains some data blocks in the data segment, and it is considered that the data blocks are distributed non-sequentially in the container.

[0061] In the present invention, for sequentially distributed data blocks, data deduplication technology is used for processing, by identifying and eliminating duplicate data blocks, only retaining unique copies, thereby reducing write operations, improving compression rate, and reducing write time and hard disk space consumption, thereby improving recovery performance. For non-sequentially distributed data blocks, they are rewritten into corresponding containers to ensure the order and continuity of the data blocks, and avoid the degradation of recovery performance due to data block fragmentation.

[0062] The present invention adopts data deduplication technology to reduce the writing operation of data blocks in sequential distribution containers, thereby improving the compression rate and effectively reducing the consumption of writing time and hard disk space. For data blocks in non-sequential distribution containers, the hard disk needs to be written again to maintain the order of the data blocks, thereby improving data recovery performance and making the distribution of data on the hard disk more continuous. This optimization scheme not only improves storage efficiency and recovery speed, but also reduces hard disk wear and saves hard disk space.

[0063] Based on the above embodiment, the local joint redundancy elimination process is performed on other data blocks except the first data block in the container in the data backup system, including: Performing data deduplication processing on the second data blocks in each container in the data backup system, wherein the second data blocks are data blocks in the data segment other than the first data blocks and determined to be duplicate data; According to the characteristic value of the third data block, a corresponding reference data block is matched from the data backup system, and if the third data block is determined to be a similar data block based on the reference data block, differential compression processing is performed on the third data block; The third data block is a data block in the data segment except the first data block and the second data block.

[0064] In the present invention, multi-container data deduplication processing and differential compression are used to further eliminate redundant data, that is, local joint redundancy elimination. The processing process is to eliminate completely repeated data blocks in multiple containers through data deduplication, and then eliminate redundant data in similar but non-duplicate data blocks through differential compression.

[0065] In the present invention, local deduplication (i.e., multi-container data deduplication processing) is part of the local joint deduplication process, which is similar to the sequential deduplication in the above embodiment, but the process can perform deduplication in more containers. Sequential deduplication only deduplicates containers referenced by a single segment, while local deduplication can not only deduplicate these containers, but also deduplicate all containers written since the last sequential deduplication.

[0066] Local differential compression (i.e. differential compression processing) can calculate the eigenvalues ​​of data blocks that have not been locally deduplicated and detect their reference blocks by matching the eigenvalues. In order to avoid a significant impact on recovery performance, the reference blocks of differential compression are only detected from containers that have been deduplicated during data deduplication.

[0067] On the basis of the above embodiment, the method of matching and obtaining a corresponding reference data block from the data backup system according to the characteristic value of the third data block, and performing differential compression processing on the third data block when it is determined that the third data block is a similar data block based on the reference data block, includes: Based on a rolling hash algorithm, calculating multiple hash values ​​corresponding to the third data block; Based on a preset feature value extraction algorithm, determining a feature value of the third data block from the multiple hash values; According to the characteristic value of the third data block, a plurality of target historical data blocks are matched from the data backup system, wherein the target historical data block is a historical data block in a historical characteristic value table of the data backup system having the same characteristic value as the characteristic value of the third data block, and the historical characteristic value table is constructed by characteristic values ​​corresponding to the historical data blocks; Determine the target historical data block with the largest number of matches with the feature value of the third data block among the multiple target historical data blocks as the reference data block; Determine whether there is a difference between the reference data block and the third data block, and if there is a difference, determine that the third data block is a similar data block; The third data block is subjected to differential compression processing.

[0068] In the present invention, the hash value of each window is calculated one by one by a rolling hash algorithm, thereby obtaining a series of continuous hash values ​​of the third data block, and these hash values ​​represent the characteristics of different parts of the data block. Furthermore, the characteristic value of the third data block is selected from the multiple hash values ​​calculated above. In the present invention, the preset characteristic value extraction algorithm includes Odess, NTransform or Top K algorithms, and each algorithm has its own specific way to select characteristic values. For example, the Top K algorithm may select the first K maximum values ​​in the hash value as the characteristic value; while Odess or NTransform selects the characteristic value based on a specific mathematical transformation or optimization strategy.

[0069] Furthermore, the historical characteristic value table maintained in the data backup system is used to search for historical data blocks with the same characteristic value as the third data block. In the present invention, the historical characteristic value table is a data structure that records the characteristic values ​​of all previously processed data blocks and related information thereof. By searching for historical data blocks that match the characteristic value of the third data block in this table, all data blocks that may be similar or duplicated with the third data block can be found, and these found data blocks are referred to as target historical data blocks.

[0070] Further, one is selected from a plurality of target historical data blocks as a reference data block. The selection criterion is the one among these target historical data blocks that matches the third data block feature value the most times. This reference data block will be used for subsequent comparison and differential compression with the third data block. In the present invention, more matching times indicate that the two data blocks are more similar to some extent, and therefore it is more likely to reduce storage requirements through differential compression. When there is a difference between the third data block and the reference data block, it can be determined that the third data block is a similar data block, rather than a completely duplicated data block.

[0071] Finally, after determining that the third data block is a similar data block, differential compression is performed on the third data block, and only the difference parts (i.e., the differentials) and the reference to the reference data block can be stored instead of the entire third data block. When the third data block needs to be restored, the differentials can be applied to the reference data block to reconstruct the original third data block, thereby significantly reducing storage requirements and improving storage efficiency.

[0072] In one embodiment, the total redundancy caused by the redundancy between the current data block (i.e., the third data block) and the reference data block is estimated, and the total redundancy is accumulated, and then the result of the total redundancy accumulation is divided by the size of the data set, thereby obtaining the differential compression rate estimation value. Next, the characteristic value of the current data block is inserted into the historical characteristic value table, and the differential compression rate of the current data block is calculated based on the information obtained by searching for similar blocks, and then the differential compression rate estimation value is compensated to the differential compression rate of the current data block, and accumulated to the overall differential compression rate, thereby performing a performance evaluation on the differential compression result of this time.

[0073] In one embodiment, a similar data compression processing method based on recovery perception is provided, and the specific steps include: Step 1: When backing up data, simulate a recovery cache, and after the backup is completed, calculate the recovery factor of this backup and write it into a file; when backing up the data next time, read the file and decide which redundancy elimination strategy to use for the next data backup based on the recovery factor; Step 2: After obtaining the data stream of this data backup, divide the data stream into multiple data segments and set a queue for each data segment; then, check whether the data storage on the hard disk is distributed sequentially, and perform data deduplication processing on the data blocks in the sequential distribution container; write the data blocks in the non-sequential distribution container to the hard disk again to maintain the order of the data blocks; Step three: Use data deduplication to eliminate completely duplicate data blocks in multiple containers, and then use differential compression to eliminate redundant data in similar but non-duplicate data blocks.

[0074] The present invention can significantly reduce the amount of data transmission under the premise of ensuring a certain accuracy through compressed sensing technology. The compressed sensing technology can only sample and compress a part of the signal, and then restore the original signal through an optimization algorithm, thereby reducing the redundancy of the data. In addition, the original data or approximate original data can be effectively restored from the compressed data when needed, ensuring the integrity and availability of the data. Since the data is efficiently compressed, the method can significantly reduce the use of storage space and the demand for transmission bandwidth, which is particularly important in processing large-scale data sets or scenarios where frequent data transmission is required. It can significantly reduce costs and improve the performance of the overall data backup system. Furthermore, the present invention can further eliminate redundant parts in the data through technologies such as sequential deduplication and local joint redundancy elimination, thereby improving the efficiency of data processing. In addition, the present invention simulates the recovery process during backup and evaluates the recovery performance, thereby optimizing the data recovery process.

[0075] The data compression processing system provided by the present invention is described below. The data compression processing system described below and the data compression processing method described above can be referenced to each other.

[0076] Figure 2 The structural diagram of the data compression processing system provided by the present invention is as follows: Figure 2 As shown, the present invention provides a data compression processing system, including a data segmentation module 201, a data deduplication object selection module 202 and a local joint redundancy elimination module 203, wherein the data segmentation module 201 is used to obtain a backup data stream corresponding to a current data compression processing task, and perform data segmentation processing on the backup data stream to obtain multiple data segments; the data deduplication object selection module 202 is used to write the multiple data segments into a data backup system, and if it is determined that the data blocks in the data segment are sequentially distributed in the container in the data backup system according to the container reference rate of the container in the data backup system, the first data block is a data block in which the data blocks in the data segment are sequentially distributed in the container in the data backup system; the local joint redundancy elimination module 203 is used to perform local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block, so as to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0077] On the basis of the above embodiment, the system also includes a recovery perception module, and the recovery perception module includes a data recovery unit. The data recovery unit is specifically used to: read the data blocks in the corresponding container in the data backup system according to the sequence of data blocks in the file spectrum corresponding to the current data compression processing task; if the read data block is a data block after differential compression processing, obtain the corresponding reference data block, and according to the reference data block, perform differential decoding processing on the data block after differential compression processing to obtain the data block after data recovery.

[0078] Based on the above embodiment, the recovery perception module also includes a write-time simulation recovery unit, which is used to determine the corresponding container number in the data backup system according to the data block type of each data block in the multiple data segments during the process of writing the multiple data segments into the data backup system, wherein the data block type includes duplicate data blocks, differential blocks, reference blocks, and non-duplicate and non-differential compressed data blocks; according to the container number, simulate the recovery cache state during the data recovery process.

[0079] On the basis of the above embodiment, the local joint redundancy elimination module includes a local data deduplication unit and a local differential compression unit. The local data deduplication unit is used to perform data deduplication processing on the second data block in each container in the data backup system, wherein the second data block is a data block in the data segment other than the first data block and determined to be duplicate data; the local differential compression unit is used to match the corresponding reference data block from the data backup system according to the characteristic value of the third data block, and if based on the reference data block, when the third data block is determined to be a similar data block, differential compression processing is performed on the third data block; wherein the third data block is a data block in the data segment other than the first data block and the second data block.

[0080] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0081] Figure 3 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor (Processor) 301, a communication interface (Communications Interface) 302, a memory (Memory) 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304. The processor 301 can call the logic instructions in the memory 303 to execute a data compression processing method, which includes: obtaining a backup data stream corresponding to a current data compression processing task, and performing data segmentation processing on the backup data stream to obtain multiple data segments; writing the multiple data segments into a data backup system, and if it is determined that the data blocks in the data segment are sequentially distributed in the container in the data backup system according to the container reference rate of the container in the data backup system, performing data deduplication processing on the first data block, wherein the first data block is a data block in the data segment that is sequentially distributed in the container in the data backup system; performing local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0082] In addition, the logic instructions in the above-mentioned memory 303 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data compression processing methods provided by the above methods, and the method includes: obtaining a backup data stream corresponding to a current data compression processing task, and performing data segmentation processing on the backup data stream to obtain multiple data segments; writing the multiple data segments into a data backup system, and if it is determined that the data blocks in the data segment are sequentially distributed in the container in the data backup system according to the container reference rate of the container in the data backup system, performing data deduplication processing on the first data block, wherein the first data block is a data block in which the data blocks in the data segment are sequentially distributed in the container in the data backup system; performing local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0084] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the data compression processing method provided by the above-mentioned embodiments, the method comprising: obtaining a backup data stream corresponding to a current data compression processing task, and performing data segmentation processing on the backup data stream to obtain multiple data segments; writing the multiple data segments into a data backup system, and if it is determined that the data blocks in the data segment are sequentially distributed in the container in the data backup system according to the container reference rate of the container in the data backup system, performing data deduplication processing on the first data block, wherein the first data block is a data block in the data segment that is sequentially distributed in the container in the data backup system; performing local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression processing method, characterized in that: include: Acquire a backup data stream corresponding to the current data compression processing task, and perform data segmentation processing on the backup data stream to obtain multiple data segments; Writing the multiple data segments into a data backup system, and if it is determined, based on a container reference rate of a container in the data backup system, that data blocks in the data segment are sequentially distributed in a container in the data backup system, performing data deduplication processing on a first data block, wherein the first data block is a data block in which data blocks in the data segment are sequentially distributed in a container in the data backup system; Local joint redundancy elimination processing is performed on other data blocks in the container of the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

2. The data compression processing method according to claim 1, characterized in that: After performing local joint redundancy elimination processing on other data blocks except the first data block in the container in the data backup system to complete the current data compression processing task, the method further includes: Reading the data blocks in the corresponding container in the data backup system according to the sequence of the data blocks in the file spectrum corresponding to the current data compression processing task; If the read data block is a data block after differential compression processing, obtain the corresponding reference data block, and perform differential decoding processing on the data block after differential compression processing according to the reference data block to obtain a data block after data recovery.

3. The data compression processing method according to claim 2, characterized in that: The method further comprises: In the process of writing the plurality of data segments into the data backup system, determining a corresponding container number in the data backup system according to a data block type of each data block in the plurality of data segments, wherein the data block type includes a duplicate data block, a differential block, a reference block, and a non-duplicate and non-differentially compressed data block; According to the container number, a recovery cache state during data recovery is simulated.

4. The data compression processing method according to claim 2, characterized in that: If the read data block is a data block after differential compression processing, the corresponding reference data block is obtained, and according to the reference data block, the differential compression data block is differentially decoded to obtain the data block after data recovery, the method further includes: Calculate the recovery factor according to the data size of the data block after the data is recovered and the corresponding container read quantity; According to the mapping relationship between the recovery factor and the redundancy elimination strategy, a target redundancy elimination strategy corresponding to the next data compression processing task is obtained.

5. The data compression processing method according to claim 1, characterized in that: If it is determined according to the container reference rate of the container in the data backup system that the data blocks in the data segment are sequentially distributed in the container in the data backup system, performing data deduplication processing on the first data block includes: Obtaining the total size of data blocks in the data segment written into the container; Calculating the container reference rate corresponding to the container according to the total size of the data blocks and the total storage space of the container; According to the container reference rate and a preset reference rate threshold, determining whether all data blocks in the container are data blocks in the data segment; If all the data blocks in the container are data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is sequential distribution; Perform data deduplication processing on sequentially distributed data blocks; If the data blocks in the container are part of the data blocks in the data segment, determining that the distribution mode of the data blocks in the data segment in the container in the data backup system is non-sequential distribution; Write the non-sequentially distributed data blocks to the corresponding container.

6. The data compression processing method according to claim 1, characterized in that: The locally joint redundancy elimination process is performed on the other data blocks except the first data block in the container in the data backup system, including: Performing data deduplication processing on the second data blocks in each container in the data backup system, wherein the second data blocks are data blocks in the data segment other than the first data blocks and determined to be duplicate data; According to the characteristic value of the third data block, a corresponding reference data block is matched from the data backup system, and if the third data block is determined to be a similar data block based on the reference data block, differential compression processing is performed on the third data block; The third data block is a data block in the data segment except the first data block and the second data block.

7. The data compression processing method according to claim 6, characterized in that: The method of obtaining a corresponding reference data block from the data backup system according to the characteristic value of the third data block, and performing differential compression processing on the third data block when the third data block is determined to be a similar data block based on the reference data block, includes: Based on a rolling hash algorithm, calculating multiple hash values ​​corresponding to the third data block; Based on a preset feature value extraction algorithm, determining a feature value of the third data block from the multiple hash values; According to the characteristic value of the third data block, a plurality of target historical data blocks are matched from the data backup system, wherein the target historical data block is a historical data block in a historical characteristic value table of the data backup system having the same characteristic value as the characteristic value of the third data block, and the historical characteristic value table is constructed by characteristic values ​​corresponding to the historical data blocks; Determine the target historical data block with the largest number of matches with the feature value of the third data block among the multiple target historical data blocks as the reference data block; Determine whether there is a difference between the reference data block and the third data block, and if there is a difference, determine that the third data block is a similar data block; The third data block is subjected to differential compression processing.

8. A data compression processing system, characterized in that: include: A data segmentation module is used to obtain a backup data stream corresponding to the current data compression processing task, and perform data segmentation processing on the backup data stream to obtain multiple data segments; a data deduplication object selection module, configured to write the plurality of data segments into a data backup system, and if it is determined, based on a container reference rate of a container in the data backup system, that data blocks in the data segment are sequentially distributed in a container in the data backup system, perform data deduplication processing on a first data block, wherein the first data block is a data block in which data blocks in the data segment are sequentially distributed in a container in the data backup system; A local joint redundancy elimination module is used to perform local joint redundancy elimination processing on other data blocks in the container in the data backup system except the first data block to complete the current data compression processing task, wherein the local joint redundancy elimination processing includes multi-container data deduplication processing and differential compression processing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data compression processing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data compression processing method according to any one of claims 1 to 7 is implemented.

Images