Data compression method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data compression method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that an initial data block group is determined according to a plurality of to-be-compressed data blocks, the initial data block group comprises a plurality of target data blocks, and the data similarity between the target data blocks is larger than a preset threshold value; determining a first prediction compression ratio corresponding to the initial data block group according to the initial compression ratio of each target data block; according to the first prediction compression ratio, a target data block group corresponding to the initial data block group is obtained, merging compression processing is conducted on the target data block group, and a second prediction compression ratio corresponding to the target data block group is larger than or equal to a preset compression ratio threshold value. By adopting the method, the data compression efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data compression, and in particular to a data compression method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] As the amount of data to be stored increases, more storage space is needed to store the corresponding data such as photos, videos, documents, and applications. By removing redundant information and optimizing the storage format through data compression, the compressed data can occupy less storage space.

[0003] Typically, data compression is performed by merging and compressing multiple data blocks together and then compressing them to obtain a higher compression rate than compressing a single data block. However, the traditional technology has the problem of relatively low data compression efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data compression method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the efficiency of data compression in order to address the above technical problems.

[0005] In a first aspect, the present application provides a data compression method. The method comprises:

[0006] Determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0007] Determining a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0008] According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0009] In one embodiment, obtaining a target data block group corresponding to the initial data block group according to the first predicted compression rate includes:

[0010] If the first predicted compression rate is less than the compression rate threshold, updating the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group;

[0011] If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0012] In one embodiment, the updating of the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group includes:

[0013] The initial data block group is used as the current data block group, and at least one data block update is performed. For each data block update process, a replacement data block is determined from the other data blocks, and the current data block group is updated using the replacement data block until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

[0014] In one embodiment, the step of determining a replacement data block from the other data blocks and updating the current data block group with the replacement data block until the target data block group is obtained includes:

[0015] Determine the replacement data block from the other data blocks, and determine the data block compression ratio of each of the target data blocks included in the current data block group;

[0016] Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group;

[0017] If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0018] In one embodiment, the method further comprises:

[0019] If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0020] In one embodiment, before taking the initial data block group as the current data block group and performing at least one data block update, the method further includes:

[0021] Determine whether the traversal of each of the data blocks is completed;

[0022] The step of taking the initial data block group as the current data block group and performing at least one data block update comprises:

[0023] If the traversal of the data blocks has not been completed, the initial data block group is used as the current data block group, and at least one data block update is performed.

[0024] In one embodiment, the method further comprises:

[0025] If the traversal of the data blocks is completed, the initial data block group is merged and compressed.

[0026] In one embodiment, after merging and compressing the target data block group, the method further includes:

[0027] Obtaining an actual compression ratio corresponding to the target data block group;

[0028] If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

[0029] In one embodiment, the method further comprises:

[0030] Determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process;

[0031] If the number of the data blocks is less than a preset number threshold, decompressing the compressed data blocks to obtain a plurality of decompressed data blocks;

[0032] A new target data block group is determined according to the multiple decompressed data blocks and other data blocks to be compressed, and the new target data block group is merged and compressed.

[0033] In a second aspect, the present application also provides a data compression device. The device comprises:

[0034] A first determination module, configured to determine an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0035] A second determination module, configured to determine a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0036] The first compression module is used to obtain a target data block group corresponding to the initial data block group according to the first predicted compression rate, and merge and compress the target data block group, wherein the second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0039] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0040] In the above-mentioned data compression method, apparatus, computer device, computer-readable storage medium and computer program product, the server determines an initial data block group according to a plurality of data blocks to be compressed, the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold value, then, according to the initial compression rate of each target data block, a first predicted compression rate corresponding to the initial data block group is determined, then, according to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold value. Threshold. In the data compression method, before the server performs the merge compression process, it can first obtain the first predicted compression rate corresponding to the initial data block group, and obtain the target data block group greater than or equal to the compression rate threshold according to the first predicted compression rate, so that the second predicted compression rate corresponding to the target data block group can meet the expected compression rate, so that after the target data block group is merged and compressed, the compression rate after the merge compression process is greater than or equal to the compression rate threshold, avoiding the problem of read amplification caused by too low compression rate, or invalid compression caused by too high compression rate, improving the success rate of the merge compression process, and thus improving the efficiency of data compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] FIG1 is a diagram showing an application environment of a data compression method according to an embodiment;

[0043] FIG2 is a schematic diagram of a flow chart of a data compression method in one embodiment;

[0044] FIG3 is a schematic diagram of obtaining a first predicted compression rate corresponding to an initial data block group in another embodiment;

[0045] FIG4 is a schematic flow chart of a data compression method in another embodiment;

[0046] FIG5 is a schematic flow chart of a data compression method in another embodiment;

[0047] FIG6 is a schematic flow chart of a data compression method in another embodiment;

[0048] FIG7 is a schematic diagram of merging and compressing a new target data block group in one embodiment;

[0049] FIG8 is a schematic flow chart of a data compression method in another embodiment;

[0050] FIG9 is a block diagram of a data compression device in one embodiment;

[0051] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In the field of merge compression, data is usually compressed in the form of background compression, that is, the server obtains the data block to be compressed from the disk, and then groups the data blocks according to the similarity between the data blocks. Next, a group of data blocks with high similarity are merged together and compressed again, and then the compressed data blocks obtained after compression are persisted to the disk. Usually, the size of a single data block is 8KB, and the number of a group of similar data blocks can be 16. The size of the compressed data block obtained after merge compression is different due to the different compression ratios selected. For example, if the compression ratio is 1.6, the size of the compressed data block after merge compression is 80KB. When it is necessary to read a data block in the compressed data block, it is necessary to read a compressed data block first, that is, in order to read 8KB of data, it is necessary to read 80KB of data first. Therefore, it will cause the problem of read amplification and affect the read performance of the server.

[0054] In order to solve the problem of read amplification, a minimum compression rate threshold can be set. For example, the compression rate can be 4, and the size of the compressed data block after the merge compression is 32KB. When reading data, the amount of data obtained can be effectively reduced, improving the read amplification problem. However, in this method, if the compression rate of the compressed data block after the merge compression is less than the minimum compression rate threshold, the merge compression is considered to be an invalid data compression process, and the compressed data block needs to be discarded. Therefore, this method has the problem of wasting system resources of the server.

[0055] In view of this, in the data compression method provided by the embodiment of the present application, the server determines an initial data block group according to multiple data blocks to be compressed, the initial data block group includes multiple target data blocks, and the data similarity between each target data block is greater than a preset threshold value, then, according to the initial compression rate of each target data block, the first predicted compression rate corresponding to the initial data block group is determined, and then, according to the first predicted compression rate, the target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and the second predicted compression rate corresponding to the target data block group is greater than or equal to the preset compression rate threshold value. In the data compression method, before the server performs the merge compression process, it can first obtain the first predicted compression rate corresponding to the initial data block group, and obtain the target data block group greater than or equal to the compression rate threshold value according to the first predicted compression rate, so that the second predicted compression rate corresponding to the target data block group can meet the expected compression rate, so that after the target data block group is merged and compressed, the compression rate after the merge compression process is greater than or equal to the compression rate threshold value, avoiding the problem of read amplification caused by too low compression rate, or invalid compression caused by too high compression rate, improving the success rate of merge compression processing, and thus improving the efficiency of data compression.

[0056] The data compression method provided in the embodiment of the present application can be applied in an implementation environment as shown in Figure 1, which includes a server. The server can be implemented by an independent server or a server cluster composed of multiple servers, and the data storage system can store the data that the server needs to process. The data storage system can be integrated on the server, or it can be placed on the cloud or other network servers. Among them, the server can determine an initial data block group including multiple target data blocks based on multiple data blocks to be compressed, and the data similarity between each target data block is greater than a preset threshold. Then, according to the initial compression rate of each target data block, the first predicted compression rate corresponding to the initial data block group is determined, and then, according to the first predicted compression rate, the target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed. The second predicted compression rate corresponding to the target data block group is greater than or equal to the preset compression rate threshold.

[0057] In other possible implementations, the data compression method provided in the embodiment of the present application can also be applied to terminals, which can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0058] In an exemplary embodiment, as shown in FIG2 , a data compression method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps 201 to 203:

[0059] Step 201 : determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold.

[0060] It should be noted that data can be managed and stored on the disk in units of data blocks. During the data writing process, the server's foreground program can write the data into multiple data blocks and persist the multiple data blocks to the disk. Then, the server's background program can obtain multiple data blocks from the disk, merge the data blocks, and then perform data compression on the merged data blocks to reduce the storage space occupied by the data, that is, merging and compression.

[0061] The data blocks to be compressed refer to the data blocks in the disk that need to be compressed. In this embodiment, multiple data blocks can be selected according to the data similarity between each data block, and then the selected data blocks are determined as target data blocks, and the multiple target data blocks are determined as the initial data block group. Optionally, based on the consideration of the compression rate of the compressed data block, the number of target data blocks in the initial data block group can be limited. For example, the number of target data blocks in the initial data block group can be 16, or it can also be 14. This embodiment does not limit this. It can be understood that if there are multiple target data blocks whose data similarity is greater than a preset threshold among the multiple data blocks to be compressed, multiple initial data block groups can be obtained, wherein the preset threshold refers to the minimum value of the data similarity between the data blocks.

[0062] In this embodiment, the server can obtain multiple data blocks to be compressed from the disk, and then calculate the data feature value of each data block according to the similarity algorithm, and then compare the data similarity between each data block and other data blocks, and then determine the data block whose data similarity is greater than a preset threshold as the target data block, and determine the obtained multiple target data blocks as the initial data block group.

[0063] Step 202: Determine a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block.

[0064] The initial compression rate refers to the data compression rate corresponding to each target data block. It should be noted that when the server persists the data block to the disk, in order to save the space occupied by the data, the data block can be compressed, so that the compression rate corresponding to each data block, that is, the initial compression rate, can be obtained.

[0065] It should be noted that in order to avoid the problem of read amplification caused by too low a compression ratio and the problem of invalid compression caused by too high a compression ratio, before performing the merge compression process, the compression ratio of the initial data block group can be predicted, so as to determine whether to merge and compress the initial data block group according to the predicted compression ratio. In this embodiment, the first predicted compression ratio refers to the compression ratio obtained by predicting the compression ratio of the initial data block group. The compression ratio refers to the ratio between the size of the data before compression and the size of the data after compression, and the larger the compression ratio, the better the compression effect.

[0066] In this embodiment, the server can obtain the initial compression rate and the compressed data block size of each target data block from the disk, and then calculate the data block size of each target data block before compression based on the initial compression rate of each target data block. Then, the data block size of each target data block before compression is substituted into the compression rate calculation formula, and the calculation result is determined as the first predicted compression rate.

[0067] Exemplarily, the calculation process of the first predicted compression rate can be shown in FIG3 , where the initial data block group includes 16 target data blocks in total, and the first predicted compression rate corresponding to the initial data block group can be obtained according to the initial compression rate of each target data block.

[0068] Step 203: According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed. The second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0069] Among them, the predicted compression rate corresponding to the target data block group is the second predicted compression rate, the target data block group refers to the initial data block group that meets the preset compression rate threshold, and the compression rate threshold refers to the minimum value that can meet the compression expectations. Therefore, the second predicted compression rate is greater than or equal to the compression rate threshold.

[0070] Generally, the compression rate after the merged compression process is positively correlated with the predicted compression rate, that is, if the predicted compression rate is greater than or equal to the preset compression rate threshold, the compression rate after the merged compression process is also greater than or equal to the compression rate threshold. Therefore, if the predicted compression rate corresponding to the initial data block group is greater than or equal to the compression rate threshold, it can be said that the compression rate of the compressed data blocks after the merged compression process of the initial data block group can also be greater than or equal to the compression rate threshold, and the initial data block group can be directly merged and compressed; if the predicted compression rate corresponding to the initial data block group is less than the compression rate threshold, it can be said that the compression rate of the compressed data blocks after the merged compression process of the initial data block group is also less than the compression rate threshold, and the data blocks in the initial data block group can be adjusted, and then the adjusted initial data block group is used as the target data block group, and the target data block group is merged and compressed.

[0071] In this embodiment, the server may compare the first predicted compression rate with the compression rate threshold, and then obtain the corresponding target data block group according to the comparison result and the initial data block group, and then merge and compress the target data block group.

[0072] In the above data compression method, the server determines an initial data block group based on a plurality of data blocks to be compressed, the initial data block group includes a plurality of target data blocks, and the data similarity between each target data block is greater than a preset threshold value, then, according to the initial compression rate of each target data block, a first predicted compression rate corresponding to the initial data block group is determined, then, according to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold value. In the data compression method, before the server performs the merge compression process, it can first obtain the first predicted compression rate corresponding to the initial data block group, and obtain a target data block group greater than or equal to the compression rate threshold value based on the first predicted compression rate, so that the second predicted compression rate corresponding to the target data block group can meet the expected compression rate, so that after the target data block group is merged and compressed, the compression rate after the merge compression process is greater than or equal to the compression rate threshold value, avoiding the problem of read amplification caused by too low compression rate, or invalid compression caused by too high compression rate, improving the success rate of merge compression processing, and thus improving the efficiency of data compression.

[0073] In one embodiment, based on the above embodiment, this embodiment relates to a process of how the server obtains the target data block group corresponding to the initial data block group according to the first predicted compression rate. Two possible implementation methods of implementing step 203 are described below.

[0074] In a possible implementation of step 203, if the first predicted compression rate is less than the compression rate threshold, the initial data block group is updated according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group.

[0075] Among them, other data blocks refer to data blocks other than the target data blocks among multiple data blocks to be compressed. It can be understood that the compression rate corresponding to the initial data block group is closely related to the compression rate of each data block in the initial data block group. Therefore, when the first predicted compression rate is less than the compression rate threshold, in order to ensure the success rate and efficiency of the merged compression, the data blocks in the target data block group for merged compression can be updated so that the first predicted compression rate corresponding to the initial data block group is greater than or equal to the compression rate threshold, thereby obtaining the target data block group.

[0076] In this embodiment, when the first predicted compression rate is less than the compression rate threshold, the server can filter out data blocks having data similarity with the target data block greater than a preset threshold from other data blocks based on the data feature values ​​of the data blocks, and then use the filtered data blocks to replace the data blocks in the target data block group, thereby using the replaced data blocks as the target data block group.

[0077] Optionally, the filtered data blocks may be used to replace data blocks in one target data block group; or, the filtered data blocks may be used to replace data blocks in two target data block groups.

[0078] In another possible implementation of step 203, if the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0079] It is understandable that when the first predicted compression rate is greater than or equal to the compression rate threshold, it can be explained that after the initial data block group is merged and compressed, its compression rate can meet the preset compression expectation, and therefore, there is no need to update the data blocks in the initial data block group.

[0080] In this embodiment, when the first predicted compression rate is greater than or equal to the compression rate threshold, the server may determine the initial data block group as the target data block group.

[0081] In this embodiment, when the first predicted compression rate is less than the compression rate threshold, the server updates the initial data block group based on other data blocks in the multiple data blocks except the target data blocks to obtain the target data block group, and when the first predicted compression rate is greater than or equal to the compression rate threshold, the server determines the initial data block group as the target data block group, so that the target data block group that meets the compression expectation can be obtained quickly and accurately based on the first predicted compression rate and the compression rate threshold, thereby improving the efficiency and accuracy of obtaining the target data block group.

[0082] In one embodiment, based on the above embodiment, this embodiment relates to a process of how the server updates the initial data block group according to other data blocks except each target data block in multiple data blocks. The specific implementation process of this process can be: taking the initial data block group as the current data block group, performing at least one data block update, for each data block update process, determining a replacement data block from other data blocks, and using the replacement data block to update the current data block group, until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than a preset threshold.

[0083] It is understandable that, in the process of updating the initial data block group according to other data blocks in a plurality of data blocks except each target data block, after one data block update, there may be a situation where the compression rate of the updated data block group is greater than or equal to the compression rate threshold, in which case the updated data block group can be used as the target data block group; or, there may be a situation where the compression rate of the updated data block group is less than the compression rate threshold, in which case the updated data block group can be updated again. Therefore, in this embodiment, the server can use the initial data block group as the current data block group and perform at least one data block update. It should be noted that the current data block group is variable. When the data blocks are updated for the first time, the initial data block group is the current data block group. When the data blocks are updated for the second time, the current data block group is the data block group obtained after the first data block update, and so on, until the target data block group is obtained.

[0084] Since merging and compression refers to merging and compressing multiple data blocks whose data similarity is greater than a preset threshold, in order to improve the efficiency of data compression, data blocks whose data similarity with each target data block is greater than a preset threshold can be selected from other data blocks as replacement data blocks. The following takes the process of updating a data block of the current data block group as an example to describe the specific process of updating a data block in detail.

[0085] In this embodiment, the data similarity between each other data block and each target data block in the current data block group can be calculated respectively, and then other data blocks corresponding to data similarities greater than a preset threshold are determined as replacement data blocks, and a target data block in the current data block group is updated using the replacement data block to obtain an updated data block group, which is a new current data block group. Then, the predicted compression rate of the new current data block group is calculated. If the predicted compression rate is greater than or equal to the compression rate threshold, the new current data block group is determined as the target data block group. Otherwise, if the predicted compression rate is less than the compression rate threshold, a new replacement data block is reselected from other data blocks, and the above replacement process is repeated until the predicted compression rate of the new current data block group obtained in a certain update process is greater than or equal to the compression rate threshold, the new current data block group is determined as the target data block group, and the update process is stopped.

[0086] It is understandable that when updating the initial data block group, it is necessary to select data blocks from other data blocks to update the target data blocks in the initial data block group. Then, when there are no other data blocks, it is impossible to obtain the replacement data blocks. Therefore, in a possible implementation, the server can determine whether the traversal of each data block is completed before using the initial data block group as the current data block group and performing at least one data block update. In this implementation, the server can obtain the operation record corresponding to each data block, and then determine from the operation record whether each data block has replaced the data block in the current data block group, thereby determining whether the traversal of each data block is completed.

[0087] On this basis, the above-mentioned method of using the initial data block group as the current data block group and performing at least one data block update includes: if the traversal of each data block has not been completed, using the initial data block group as the current data block group and performing at least one data block update. That is to say, if the traversal of each data block has not been completed, the server can use the initial data block group as the current data block group to perform a data block update once, and further, after the data block update, it can once again determine whether the traversal of each data block has been completed, so that if the traversal of each data block has not been completed, the current data block group can be updated again to obtain a new current data block group. In this process, the server can determine whether the traversal of each data block has been completed before performing at least one data block update, so that the data block can be updated when the traversal has not been completed, thereby ensuring the reliability of the data block update process.

[0088] It is understandable that if the traversal of each data block is completed, no new replacement data block can be obtained, and there is no need to perform a data block update again. Therefore, in a possible implementation, if the traversal of each data block is completed, the server does not need to perform a data block update process, and can directly merge and compress the initial data block group. The server can merge and compress the initial data block group when the traversal of each data block is completed, avoiding the problem of performing invalid data block update operations and improving the efficiency of the merge and compress process.

[0089] In this embodiment, the server uses the initial data block group as the current data block group and performs at least one data block update. For each data block update process, the server can determine a replacement data block from other data blocks. Since the data similarity between the replacement data block and each target data block included in the current data block group is greater than a preset threshold, the replacement data block can meet the compression requirements of the merge compression, so the current data block group is updated with the replacement data block until the target data block group is obtained, which can increase the possibility that the predicted compression rate of the target data block group is greater than or equal to the compression rate threshold, thereby improving the efficiency of obtaining the target data block group.

[0090] In one embodiment, based on the above embodiment, referring to FIG4, this embodiment relates to how the server determines a replacement data block from other data blocks, and updates the current data block group with the replacement data block until the target data block group is obtained. As shown in FIG4, the server can perform steps 401 to 403 to implement the process.

[0091] Step 401: determine a replacement data block from other data blocks, and determine a data block compression ratio of each target data block included in the current data block group.

[0092] The data block compression rate refers to the compression rate corresponding to each target data block in the current data block group. In this embodiment, the server can determine the data blocks whose data similarity between other data blocks and each target data block is greater than a preset threshold as replacement data blocks, and obtain the data block compression rate of each target data block included in the current data block group.

[0093] Step 402: Use the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group to obtain an intermediate data block group.

[0094] It can be understood that, when the predicted compression rate of the current data block group is less than the compression rate threshold, the target data block in the current data block group can be replaced. Therefore, in order to improve the predicted compression rate of the current data block group, the target data block with the smallest replacement data block compression rate in the current data block group can be replaced. The intermediate data block group refers to the data block group obtained after replacing the data blocks in the current data block group.

[0095] In this embodiment, the server may use the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, and then determine the replaced data block group as the intermediate data block group.

[0096] Step 403: If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0097] The third predicted compression rate is obtained by predicting and calculating the compression rate of the intermediate data block group. When the third predicted compression rate is greater than or equal to the compression rate threshold, it means that the compression rate of the intermediate data block group can meet the compression expectation, so the intermediate data block group can be determined as the target data block group.

[0098] In this embodiment, after obtaining the intermediate data block group, the server can calculate the third predicted compression rate corresponding to the intermediate data block group based on the initial compression rate of each data block in the intermediate data block group, and then compare the third predicted compression rate with the compression rate threshold, so as to determine the intermediate data block group as the target data block group when the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold.

[0099] As a possible implementation, if the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0100] It should be noted that if the third predicted compression rate is less than the compression rate threshold, it means that the compression rate of the intermediate data block group does not meet the compression expectation. Therefore, a new replacement data block can be obtained from other data blocks, and the intermediate data block group is used as the current data block group to continue to update the data block group. In another case, even if the third predicted compression rate corresponding to the intermediate data block group is less than the compression rate threshold, if the traversal of other data blocks has been completed and a new replacement data block cannot be obtained, the intermediate data block group is merged and compressed.

[0101] In this embodiment, the server determines a replacement data block from other data blocks, and determines the data block compression rate of each target data block included in the current data block group. Then, the replacement data block can be used to replace the target data block with the smallest data block compression rate in the current data block group to obtain an intermediate data block group. Since the target data block with the smallest compression rate is replaced, the third predicted compression rate corresponding to the intermediate data block group can be greatly improved, thereby increasing the possibility that the third predicted compression rate is greater than or equal to the compression rate threshold. Furthermore, when the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group can be determined as the target data block group, thereby improving the efficiency of obtaining the target data block group.

[0102] In one embodiment, based on the above embodiment, referring to FIG5, this embodiment relates to how the server updates the target data block group if the actual compression rate is lower than the compression rate threshold after merging and compressing the target data block group. As shown in FIG5, the server can execute steps 501 and 502 to implement the process.

[0103] Step 501: Obtain the actual compression ratio corresponding to the target data block group.

[0104] The actual compression ratio refers to the compression ratio obtained after the target data block group is combined and compressed. In this embodiment, the server can obtain the actual compression ratio corresponding to the target data block group after combining and compressing the target data block group.

[0105] Step 502: if the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

[0106] It should be noted that, usually, the actual compression rate after the merge compression processing is positively correlated with the predicted compression rate, that is, if the predicted compression rate is greater than or equal to the compression rate threshold, the actual compression rate is also greater than or equal to the compression rate threshold. However, there may be a situation where the predicted compression rate is greater than or equal to the compression rate threshold, but the actual compression rate is less than the compression rate threshold. Therefore, in order to minimize the impact of invalid merge compression due to the actual compression rate not meeting the standard, the target data block with the smallest data block compression rate can be deleted from the target data block group.

[0107] In this embodiment, when the actual compression rate is less than the compression rate threshold, the server can determine the target data block with the smallest compression rate according to the compression rates corresponding to the target data blocks in the target data block group, and then delete the target data block. After that, the server can re-merge and compress the deleted target data block group.

[0108] As a possible implementation, after the target data block with the smallest data block compression rate is deleted from the target data block group, since the number of data blocks in the target data block group does not meet the compression requirements, it can be merged and compressed again with new data blocks to be compressed during the next round of merging and compression process of the server.

[0109] In this embodiment, by obtaining the actual compression rate corresponding to the target data block group, the server can delete the target data block with the smallest data block compression rate from the target data block group when the actual compression rate is less than the compression rate threshold, and obtain an updated data block group. Since the target data block with the smallest compression rate is deleted, the updated data block group is merged and compressed, which can reduce the impact of the actual compression rate not meeting the standard when the actual compression rate does not meet the standard, thereby reducing the cost of the compression rate not meeting the standard, and further improving the efficiency of data compression.

[0110] In one embodiment, based on the above embodiment, referring to Fig. 6, this embodiment relates to a process of how the server performs a re-merging and compression process on compressed data blocks whose number of merged data blocks is less than a preset number threshold. The server can execute steps 601 to 603 to implement the process.

[0111] Step 601, determining the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process.

[0112] The number of data blocks refers to the number of data blocks in the compressed data block after the merge compression process. It should be noted that, according to the compression requirements of the merge compression, usually, the number of data blocks in the target data block group meets a certain number threshold. However, during the merge compression process, the number of data blocks in some target data block groups may be less than the number threshold. In order to reduce the space occupied by data as much as possible, the target data block group with a number of data blocks less than the number threshold can be merged and compressed with the new data block to be compressed during the next round of merge compression.

[0113] Optionally, in this embodiment, the server may obtain the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process from the file management system of the storage disk, or obtain the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process from the operation log of the system. If there are multiple compressed data blocks, the server needs to obtain the number of data blocks corresponding to each compressed data block; or, if there is one compressed data block, the server needs to obtain the number of data blocks corresponding to one compressed data block.

[0114] Step 602: If the number of data blocks is less than a preset number threshold, decompress the compressed data blocks to obtain a plurality of decompressed data blocks.

[0115] The preset number threshold refers to the number of data blocks in each target data block group. The preset number threshold can be determined based on the compression ratio after the combined compression process and the size of the storage space occupied by the compressed data blocks. For example, the preset number threshold can be 16, or the preset number threshold can also be 18, which is not limited in this embodiment.

[0116] In this embodiment, when the number of data blocks is less than a preset number threshold, the server may obtain the compressed data block from the disk, and then decompress the data block to obtain multiple data blocks from the decompression result.

[0117] Step 603: determine a new target data block group according to the multiple decompressed data blocks and other data blocks to be compressed, and merge and compress the new target data block group.

[0118] Among them, the other data blocks to be compressed are data blocks in the disk that have not been merged and compressed. In this embodiment, the server can filter out data blocks whose data similarity with multiple decompressed data blocks is greater than a preset threshold from other data blocks to be compressed, and then select a number of data blocks from the filtered data blocks according to the number threshold, and use the several data blocks and the multiple decompressed data blocks as a new data block group, and when the predicted compression rate of the new data block group is greater than or equal to the compression rate threshold, determine the new data block group as the target data block group, and merge and compress the new target data block group.

[0119] Exemplarily, the process of merging and compressing a new target data block group can be shown in FIG7 . In FIG7 , two data blocks can be obtained after decompression of the compressed data blocks, and 14 data blocks are screened out from other data blocks to be compressed. Then, the new target data block group including 16 data blocks can be merged and compressed.

[0120] In this embodiment, the server determines the number of data blocks corresponding to the compressed data blocks obtained by the merged compression processing, and can decompress the compressed data blocks when the number of data blocks is less than a preset number threshold, so as to obtain multiple decompressed data blocks, thereby being able to determine a new target data block group based on the multiple decompressed data blocks and other data blocks to be compressed, and merge and compress the new target data block group, and can re-merge and compress the compressed data blocks that do not meet the number threshold, so that the compressed data blocks can meet the number threshold, and then the compressed data blocks can meet the compression expectations, thereby improving the efficiency of data compression.

[0121] In one embodiment, a data compression method is provided for use in a server, as shown in FIG8 , and the method comprises the following steps:

[0122] Step 801: determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold.

[0123] Step 802: Determine a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block.

[0124] Step 803: determine whether the first predicted compression rate is less than the compression rate threshold.

[0125] Step 804: If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0126] Step 805 : performing a merge compression process on the target data block group, wherein the second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0127] Step 806: If the first predicted compression rate is less than the compression rate threshold, determine whether the traversal of each data block is completed.

[0128] Step 807: If the traversal of each data block is completed, the initial data block group is merged and compressed.

[0129] Step 808: If the traversal of all data blocks has not been completed, a replacement data block is determined from other data blocks.

[0130] Step 809: determine the data block compression ratio of each target data block included in the current data block group.

[0131] Step 8010: Use the replacement data block to replace the target data block with the minimum data block compression rate in the current data block group to obtain an intermediate data block group.

[0132] Step 8011, determining whether the third predicted compression rate corresponding to the intermediate data block group is less than the compression rate threshold.

[0133] Step 8012: If the third predicted compression rate is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0134] Step 8013: If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0135] Step 8014, obtaining the actual compression ratio corresponding to the target data block group.

[0136] Step 8015, determine whether the actual compression rate is less than the compression rate threshold.

[0137] Step 8016: If the actual compression rate is greater than or equal to the compression rate threshold, the compressed data block obtained by the combined compression process is stored.

[0138] Step 8017: If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group.

[0139] Step 8018, merge and compress the updated data block group.

[0140] Step 8019, determining the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process.

[0141] Step 8020, determining whether the number of data blocks corresponding to the compressed data block is less than a preset number threshold.

[0142] Step 8021: If the number of data blocks is less than a preset number threshold, decompress the compressed data blocks to obtain a plurality of decompressed data blocks.

[0143] Step 8022: determine a new target data block group based on the multiple decompressed data blocks and other data blocks to be compressed, and merge and compress the new target data block group.

[0144] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a data compression device for implementing the data compression method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data compression device embodiments provided below can refer to the limitations on the data compression method above, and will not be repeated here.

[0146] In an exemplary embodiment, as shown in FIG9 , a data compression device is provided, comprising:

[0147] A first determination module 901 is used to determine an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0148] A second determination module 902, configured to determine a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0149] The first compression module 903 is used to obtain a target data block group corresponding to the initial data block group according to the first predicted compression rate, and merge and compress the target data block group, wherein the second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0150] In one embodiment, the first compression module 903 includes:

[0151] an updating unit, configured to update the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group if the first predicted compression rate is less than the compression rate threshold;

[0152] A determining unit is configured to determine the initial data block group as the target data block group if the first predicted compression rate is greater than or equal to the compression rate threshold.

[0153] In one of the embodiments, the updating unit is specifically used to: use the initial data block group as the current data block group, perform at least one data block update, determine a replacement data block from the other data blocks for each data block update process, and use the replacement data block to update the current data block group until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

[0154] In one of the embodiments, the updating unit is specifically used to: determine the replacement data block from the other data blocks, and determine the data block compression rate of each of the target data blocks included in the current data block group;

[0155] Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group;

[0156] If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0157] In one embodiment, the device further comprises:

[0158] The processing module is used for, if the third predicted compression rate is less than the compression rate threshold, taking the intermediate data block group as the current data block group to continue data block update, or merging and compressing the intermediate data block group.

[0159] In one embodiment, the device further comprises:

[0160] The third determination module is used to determine whether the traversal of each data block is completed.

[0161] The updating unit is specifically used to: if the traversal of each data block has not been completed, use the initial data block group as the current data block group and perform at least one data block update.

[0162] In one embodiment, the device further comprises:

[0163] The second compression module is used to merge and compress the initial data block group if the traversal of each data block is completed.

[0164] In one embodiment, the device further comprises:

[0165] A first acquisition module, used to acquire an actual compression ratio corresponding to the target data block group;

[0166] The third compression module is used to delete the target data block with the smallest data block compression rate from the target data block group if the actual compression rate is less than the compression rate threshold, obtain an updated data block group, and merge and compress the updated data block group.

[0167] In one embodiment, the device further comprises:

[0168] A fourth determination module, used to determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process;

[0169] A second acquisition module is used for decompressing the compressed data block to obtain a plurality of decompressed data blocks if the number of the data blocks is less than a preset number threshold;

[0170] The fourth compression module is used to determine a new target data block group according to the multiple decompressed data blocks and other data blocks to be compressed, and merge and compress the new target data block group.

[0171] Each module in the above data compression device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a server in the form of hardware, or can be stored in a memory in a server in the form of software, so that the processor can call and execute operations corresponding to each module.

[0172] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be compressed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data compression method is implemented.

[0173] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0175] Determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0176] Determining a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0177] According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0178] In one embodiment, the processor implements the following steps when executing the computer program:

[0179] If the first predicted compression rate is less than the compression rate threshold, updating the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group;

[0180] If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0181] In one embodiment, the processor implements the following steps when executing the computer program:

[0182] The initial data block group is used as the current data block group, and at least one data block update is performed. For each data block update process, a replacement data block is determined from the other data blocks, and the current data block group is updated using the replacement data block until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

[0183] In one embodiment, the processor implements the following steps when executing the computer program:

[0184] Determine the replacement data block from the other data blocks, and determine the data block compression ratio of each of the target data blocks included in the current data block group;

[0185] Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group;

[0186] If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0187] In one embodiment, the processor implements the following steps when executing the computer program:

[0188] If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0189] In one embodiment, the processor implements the following steps when executing the computer program:

[0190] Determine whether the traversal of each of the data blocks is completed;

[0191] The step of taking the initial data block group as the current data block group and performing at least one data block update comprises:

[0192] If the traversal of the data blocks has not been completed, the initial data block group is used as the current data block group, and at least one data block update is performed.

[0193] In one embodiment, the processor implements the following steps when executing the computer program:

[0194] If the traversal of the data blocks is completed, the initial data block group is merged and compressed.

[0195] In one embodiment, the processor implements the following steps when executing the computer program:

[0196] Obtaining an actual compression ratio corresponding to the target data block group;

[0197] If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

[0198] In one embodiment, the processor implements the following steps when executing the computer program:

[0199] Determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process;

[0200] If the number of the data blocks is less than a preset number threshold, decompressing the compressed data blocks to obtain a plurality of decompressed data blocks;

[0201] A new target data block group is determined according to the multiple decompressed data blocks and other data blocks to be compressed, and the new target data block group is merged and compressed.

[0202] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0203] Determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0204] Determining a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0205] According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0207] If the first predicted compression rate is less than the compression rate threshold, updating the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group;

[0208] If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0210] The initial data block group is used as the current data block group, and at least one data block update is performed. For each data block update process, a replacement data block is determined from the other data blocks, and the current data block group is updated using the replacement data block until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0212] Determine the replacement data block from the other data blocks, and determine the data block compression ratio of each of the target data blocks included in the current data block group;

[0213] Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group;

[0214] If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0216] If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0217] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0218] Determine whether the traversal of each of the data blocks is completed;

[0219] The step of taking the initial data block group as the current data block group and performing at least one data block update comprises:

[0220] If the traversal of the data blocks has not been completed, the initial data block group is used as the current data block group, and at least one data block update is performed.

[0221] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0222] If the traversal of the data blocks is completed, the initial data block group is merged and compressed.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0224] Obtaining an actual compression ratio corresponding to the target data block group;

[0225] If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

[0226] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0227] Determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process;

[0228] If the number of the data blocks is less than a preset number threshold, decompressing the compressed data blocks to obtain a plurality of decompressed data blocks;

[0229] A new target data block group is determined according to the multiple decompressed data blocks and other data blocks to be compressed, and the new target data block group is merged and compressed.

[0230] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0231] Determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold;

[0232] Determining a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block;

[0233] According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

[0234] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0235] If the first predicted compression rate is less than the compression rate threshold, updating the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group;

[0236] If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

[0237] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0238] The initial data block group is used as the current data block group, and at least one data block update is performed. For each data block update process, a replacement data block is determined from the other data blocks, and the current data block group is updated using the replacement data block until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

[0239] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0240] Determine the replacement data block from the other data blocks, and determine the data block compression ratio of each of the target data blocks included in the current data block group;

[0241] Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group;

[0242] If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

[0243] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0244] If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

[0245] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0246] Determine whether the traversal of each of the data blocks is completed;

[0247] The step of taking the initial data block group as the current data block group and performing at least one data block update comprises:

[0248] If the traversal of the data blocks has not been completed, the initial data block group is used as the current data block group, and at least one data block update is performed.

[0249] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0250] If the traversal of the data blocks is completed, the initial data block group is merged and compressed.

[0251] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0252] Obtaining an actual compression ratio corresponding to the target data block group;

[0253] If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

[0254] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0255] Determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process;

[0256] If the number of the data blocks is less than a preset number threshold, decompressing the compressed data blocks to obtain a plurality of decompressed data blocks;

[0257] A new target data block group is determined according to the multiple decompressed data blocks and other data blocks to be compressed, and the new target data block group is merged and compressed.

[0258] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0259] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0260] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data compression method, characterized in that: The method comprises: Determining an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold; Determining a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block; According to the first predicted compression rate, a target data block group corresponding to the initial data block group is obtained, and the target data block group is merged and compressed, and a second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

2. The method according to claim 1, characterized in that The step of obtaining a target data block group corresponding to the initial data block group according to the first predicted compression rate includes: If the first predicted compression rate is less than the compression rate threshold, updating the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group; If the first predicted compression rate is greater than or equal to the compression rate threshold, the initial data block group is determined as the target data block group.

3. The method according to claim 2, characterized in that The updating of the initial data block group according to other data blocks in the plurality of data blocks except the target data blocks to obtain the target data block group includes: The initial data block group is used as the current data block group, and at least one data block update is performed. For each data block update process, a replacement data block is determined from the other data blocks, and the current data block group is updated using the replacement data block until the target data block group is obtained, and the data similarity between the replacement data block and each target data block included in the current data block group is greater than the preset threshold.

4. The method according to claim 3, characterized in that The step of determining a replacement data block from the other data blocks and updating the current data block group with the replacement data block until the target data block group is obtained includes: Determine the replacement data block from the other data blocks, and determine the data block compression ratio of each of the target data blocks included in the current data block group; Using the replacement data block to replace the target data block with the lowest data block compression rate in the current data block group, to obtain an intermediate data block group; If the third predicted compression rate corresponding to the intermediate data block group is greater than or equal to the compression rate threshold, the intermediate data block group is determined as the target data block group.

5. The method according to claim 4, characterized in that The method further comprises: If the third predicted compression rate is less than the compression rate threshold, the intermediate data block group is used as the current data block group to continue data block updating, or the intermediate data block group is merged and compressed.

6. The method according to claim 3, characterized in that Before taking the initial data block group as the current data block group and performing at least one data block update, the method further includes: Determine whether the traversal of each of the data blocks is completed; The step of taking the initial data block group as the current data block group and performing at least one data block update comprises: If the traversal of the data blocks has not been completed, the initial data block group is used as the current data block group, and at least one data block update is performed.

7. The method according to claim 6, characterized in that The method further comprises: If the traversal of the data blocks is completed, the initial data block group is merged and compressed.

8. The method according to any one of claims 1 to 7, characterized in that: After the target data block group is merged and compressed, the method further includes: Obtaining an actual compression ratio corresponding to the target data block group; If the actual compression rate is less than the compression rate threshold, the target data block with the smallest data block compression rate is deleted from the target data block group to obtain an updated data block group, and the updated data block group is merged and compressed.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Determine the number of data blocks corresponding to the compressed data blocks obtained by the combined compression process; If the number of the data blocks is less than a preset number threshold, decompressing the compressed data blocks to obtain a plurality of decompressed data blocks; A new target data block group is determined according to the multiple decompressed data blocks and other data blocks to be compressed, and the new target data block group is merged and compressed.

10. A data compression device, characterized in that: The device comprises: A first determination module, configured to determine an initial data block group according to a plurality of data blocks to be compressed, wherein the initial data block group includes a plurality of target data blocks, and the data similarity between the target data blocks is greater than a preset threshold; A second determination module, configured to determine a first predicted compression rate corresponding to the initial data block group according to the initial compression rate of each target data block; The first compression module is used to obtain a target data block group corresponding to the initial data block group according to the first predicted compression rate, and merge and compress the target data block group, wherein the second predicted compression rate corresponding to the target data block group is greater than or equal to a preset compression rate threshold.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images