A compression decision method, device, equipment and readable storage medium
By obtaining the compressible identification information of the previous data of the target data and judging its compressibility according to the compression operation time sequence, the problem of inaccurate prediction of data compression in the prior art is solved, the prediction accuracy is improved and the calculation overhead is reduced.
Patent Information
- Application Number
- CN202211658242.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-12-22
AI Technical Summary
The prior art consumes a lot of computing resources and is not accurate enough when predicting data compression, resulting in still compressing the incompressible data.
By obtaining the compressible identification information of the previous data of the target data and arranging it according to the order of the compression operation time, the compressibility of the target data is judged, the actual or estimated compression rate is calculated, and if it is less than the preset threshold, the identification information is retained or updated. to avoid compression of incompressible data.
Improve the prediction accuracy of data compression, avoid unnecessary compression of incompressible data, and reduce the overhead of compression rate prediction.
Smart Images

Figure CN115934660B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a compression decision method, device, equipment and readable storage medium. Background Art
[0002] Data compression can reduce the storage space of data and increase the logical capacity of storage devices, thereby reducing the storage and transmission costs of data. The actual data reduction effect achieved by data compression depends on the compressibility of the data to be compressed. If the compression operation is performed on incompressible data, the size of the compressed data will not become smaller. Therefore, before compressing the data, the compressibility of the data can be predicted to avoid compressing incompressible data. The current method of predicting data compressibility consumes a lot of computing resources, and the prediction is not accurate enough, resulting in the compression of incompressible data.
[0003] Therefore, how to improve the prediction accuracy of data compressibility is a problem that technical personnel in this field need to solve. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a compression decision method, device, equipment and readable storage medium to improve the prediction accuracy of data compressibility. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a compression decision method, comprising:
[0006] If a compression decision is made for any target data, identification information for identifying whether the previous data of the target data is compressible is obtained; wherein the previous data and the target data are arranged in the order of compression operation time;
[0007] If the identification information is a compressible identification, determining that the compression decision of the target data is compressible, and calculating an actual value of the compression ratio of the target data;
[0008] If the actual value of the compression ratio is less than a preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification.
[0009] Optionally, it also includes:
[0010] If the identification information is an incompressible identification, calculating an estimated value of the compression ratio of the target data;
[0011] If the estimated compression rate is less than the preset threshold, the compression decision of the target data is determined to be compressible, and the step of calculating the actual compression rate of the target data is executed; if the actual compression rate is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification.
[0012] Optionally, calculating the estimated value of the compression ratio of the target data includes:
[0013] Extracting sample data from the target data, calculating an actual value of the compression ratio of the sample data, and using the actual value of the compression ratio of the sample data as the estimated value of the compression ratio;
[0014] and / or
[0015] The information entropy of the target data is calculated, and the compression rate estimation value is calculated based on the information entropy and a preset formula; the preset formula is obtained by fitting the relationship between the information entropy of the data and the actual compression rate.
[0016] Optionally, the process of constructing the preset formula includes:
[0017] Get multiple data blocks arranged in order;
[0018] Calculate the actual compression ratio of each data block and the information entropy of each data block;
[0019] The actual compression rate of each data block and the information entropy of each data block are fitted to obtain the fitting relationship.
[0020] Optionally, fitting the actual compression rate of each data block and the information entropy of each data block includes:
[0021] Constructing the actual compression ratio of each data block into a first array according to the arrangement order of each data block;
[0022] Constructing the information entropy of each data block into a second array according to the arrangement order of each data block;
[0023] Based on the first array and the second array, the actual compression rate of each data block and the information entropy of each data block are fitted.
[0024] Optionally, it also includes:
[0025] If at least two compression ratio estimation values are obtained, an average value of the at least two compression ratio estimation values is taken as the compression ratio estimation value of the target data.
[0026] Optionally, it also includes:
[0027] If the compression ratio estimation value is not less than the preset threshold, the compression decision of the target data is determined to be incompressible, and the identification information is updated to an incompressible identification.
[0028] Optionally, it also includes:
[0029] If the previous data does not exist, the identification information is assumed to be a compressible identification.
[0030] Optionally, it also includes:
[0031] A compression decision is made on the subsequent data of the target data according to the updated identification information; wherein the previous data, the target data and the subsequent data are arranged in the order of compression operation time.
[0032] In a second aspect, the present application provides a compression decision device, comprising:
[0033] An acquisition module, for acquiring identification information for identifying whether the previous data of any target data is compressible if a compression decision is made for the target data; wherein the previous data and the target data are arranged in the order of compression operation time;
[0034] A decision module, configured to determine that the compression decision of the target data is compressible if the identification information is a compressible identification;
[0035] An updating module is used to calculate an actual value of the compression ratio of the target data; if the actual value of the compression ratio is less than a preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification.
[0036] In a third aspect, the present application provides an electronic device, including:
[0037] Memory for storing computer programs;
[0038] A processor is used to execute the computer program to implement the compression decision method disclosed above.
[0039] In a fourth aspect, the present application provides a readable storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed compression decision method when executed by a processor.
[0040] It can be seen from the above scheme that the present application provides a compression decision method, including: if a compression decision is made for any target data, then obtaining identification information for identifying whether the previous data of the target data is compressible; wherein the previous data and the target data are arranged in chronological order of compression operation time; if the identification information is a compressible identification, then determining that the compression decision of the target data is compressible, and calculating the actual value of the compression rate of the target data; if the actual value of the compression rate is less than a preset threshold, then retaining the identification information as a compressible identification; otherwise, updating the identification information to an incompressible identification.
[0041] It can be seen that when making compression decisions for certain data, the present application makes a judgment with reference to the compressibility of the data arranged in front of the data. If the data arranged in front of the data is compressible, it is considered that the current data is also compressible. At the same time, the present application further judges the actual compression rate of the current data. When the actual compression rate of the current data is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification to determine the compressibility of the data arranged behind the current data based on the updated identification information. It can be seen that the present application can determine the compression decision of the data with reference to the relationship between the compressibility of adjacent data, and predict the compressibility of subsequent data in combination with the actual compression rate of the current data, thereby improving the prediction accuracy of data compressibility and avoiding compression of incompressible data.
[0042] It should be noted that the present application takes into account that data with adjacent storage locations or adjacent read and write operation times often have similar compressibility. For example, a large file will be divided into many small blocks on a storage device, and these blocks are stored at adjacent addresses. The data at adjacent addresses have content similarity and correlation, so the compression rate of the data at adjacent addresses is similar. Therefore, after determining that the first data block is compressible data, it can be directly determined that the second data block is also compressible. For another example, a data block constituting a certain file is modified at time 1 and is modified again in the next second of time 1. Then the data operated at these two times can be considered as data with adjacent compression operation times. If it is determined that the data operated at time 1 is compressible, it can be directly determined that the data operated in the next second of time 1 is also compressible. It can be seen that the present application considers the compression similarity between data with adjacent storage locations or adjacent read and write operation times, and makes compression decisions on the current data with reference to the compressibility of the data arranged in front of the current data, without the need to estimate the compression rate. Among them, data with adjacent storage locations or adjacent read and write operation times are generally also adjacent in compression operation time. Therefore, the data arranged in order of compression operation time has spatial continuity in its storage address and temporal continuity in its read and write operation time.
[0043] Correspondingly, a compression decision device, equipment and readable storage medium provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0045] Figure 1 A flow chart of a compression decision method disclosed in this application;
[0046] Figure 2 A flow chart of another compression decision method disclosed in this application;
[0047] Figure 3 A schematic diagram showing the relationship between Shannon entropy and compression rate of the data disclosed in this application;
[0048] Figure 4 A flow chart of a method for curve fitting based on Shannon entropy and compression rate of data disclosed in this application;
[0049] Figure 5 A flow chart of another compression decision method disclosed in this application;
[0050] Figure 6 A schematic diagram of a compression decision device disclosed in this application;
[0051] Figure 7 This is a schematic diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0053] Currently, the method of predicting data compressibility consumes a lot of computing resources, and the prediction is not accurate enough, resulting in the compression of incompressible data. To this end, the present application provides a compression decision scheme that can improve the prediction accuracy of data compressibility.
[0054] See also Figure 1 As shown, the embodiment of the present application discloses a compression decision method, including:
[0055] S101. If a compression decision is made for any target data, then identification information for identifying whether the previous data of the target data is compressible is obtained.
[0056] This embodiment is used to predict the compressibility of each data arranged in the order of compression operation time, so the previous data and the target data are arranged in the order of compression operation time.
[0057] It should be noted that data with adjacent storage locations or adjacent read / write operation times often have similar compressibility, while data with large storage location differences or large read / write operation time differences often have dissimilar compressibility. For example: a large file will be divided into many small blocks on a storage device, and these blocks are stored at adjacent addresses. The data at adjacent addresses have content similarity and relevance, so the compression rate of the data at adjacent addresses is similar. Another example: a data block constituting a certain file is modified at time 1 and modified again at the next second of time 1. Then the data operated at these two times can be considered as data with adjacent compression operation times. If it is determined that the data operated at time 1 is compressible, it can be directly determined that the data operated at the next second of time 1 is also compressible. It can be seen that considering the compression similarity between data with adjacent storage locations or adjacent read / write operation times, compression decisions can be made on the data, thereby avoiding compression rate estimation. Among them, data with adjacent storage locations or adjacent read / write operation times are generally adjacent in compression operation time. Therefore, the data arranged in the order of compression operation time has spatial continuity in its storage address and temporal continuity in its read / write operation time. That is to say, the storage addresses of data arranged in the order of compression operation time are often arranged in order of size, and their read and write operation times are often also arranged in order of time.
[0058] It should be noted that all data stored in the storage device can be compressed in theory, but after compression, some data occupies a larger storage space or the occupied storage space is not effectively reduced, then such data is incompressible data; on the contrary, if the size of a data A after compression is much smaller than the size of data A, then data A can be considered compressible data. Specifically, the compression rate can be used to describe the compressibility of a data. Compression rate = compressed data size / uncompressed data size, so the smaller the compression rate, the greater the compressibility.
[0059] S102: If the identification information is a compressible identification, determine that the compression decision of the target data is compressible, and calculate the actual value of the compression rate of the target data.
[0060] S103: If the actual value of the compression ratio is less than a preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification.
[0061] The identification information can be recorded using a global variable, or a variable can be set for each data to record the compression decision of each data. If the previous data of the target data is compressible, the compression decision of the target data is determined to be compressible, and whether to update the identification information is determined by the actual compression rate of the target data. If the actual compression rate of the target data is less than a preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification. Afterwards, a compression decision can be made for the next data of the target data based on the updated identification information; the previous data, the target data, and the next data are arranged in the order of the compression operation time.
[0062] Generally, the data arranged at the first position is considered compressible data by default, so if there is no data before a certain data, the compressibility of the current data is determined by referring to the compressibility mark. Therefore, in a specific implementation, if there is no previous data, the default mark information is the compressibility mark.
[0063] In a specific implementation, if the identification information is an incompressible identification, the estimated compression rate of the target data is calculated; if the estimated compression rate is less than a preset threshold, the compression decision of the target data is determined to be compressible, and the actual compression rate of the target data is calculated; if the actual compression rate is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification. It can be seen that if the data arranged in front of the target data is incompressible, this embodiment can estimate the estimated compression rate of the target data. When the estimated compression rate is small, the current data is considered to be compressible, otherwise, the current data is considered to be incompressible. When the estimated compression rate is small, the actual compression rate of the target data can be judged to determine whether to change the identification information. Among them, judging the actual compression rate of the target data requires compressing the target data, and then setting the compression mark according to whether the actual compression rate value is lower than the threshold. Specifically, if the actual compression rate value is lower than the threshold, the compression mark is set to compressible, otherwise, the compression mark is set to incompressible.
[0064] Generally, the estimated compression rate of a data can be estimated by various methods such as sample compression rate. Therefore, in a specific implementation, the estimated compression rate of the target data is calculated, including: extracting sample data from the target data, calculating the actual compression rate value of the sample data, and using the actual compression rate value of the sample data as the estimated compression rate; and / or calculating the information entropy of the target data, and calculating the estimated compression rate based on the information entropy and a preset formula; the preset formula is obtained by fitting the relationship between the information entropy and the actual compression rate of the data. The information entropy can be Shannon entropy.
[0065] In a specific implementation, the process of constructing the preset formula includes: obtaining a plurality of data blocks arranged in order; calculating the actual compression rate of each data block and the information entropy of each data block; fitting the actual compression rate of each data block and the information entropy of each data block to obtain a fitting relationship. In a specific implementation, fitting the actual compression rate of each data block and the information entropy of each data block includes: constructing the actual compression rate of each data block into a first array according to the arrangement order of each data block; constructing the information entropy of each data block into a second array according to the arrangement order of each data block; based on the first array and the second array, fitting the actual compression rate of each data block and the information entropy of each data block. Among them, a polynomial (such as: least squares polynomial, etc.) can be used to fit the actual compression rate of each data block and the information entropy of each data block. These data blocks can be data blocks of equal size constituting the same file, or data blocks of different sizes belonging to different files.
[0066] Since the estimated compression ratio of a data can be estimated in multiple ways, if at least two estimated compression ratios of the target data are obtained, the average of the at least two estimated compression ratios can be taken as the estimated compression ratio of the target data.
[0067] In a specific implementation, if the estimated compression ratio is not less than a preset threshold, the compression decision of the target data is determined to be incompressible, and the identification information is updated to an incompressible identification to guide the compression decision of the next data of the target data. The identification information is updated to the compression decision of the target data, so that the compression decision of the data arranged after the target data is made according to the updated identification information.
[0068] It can be seen that when making compression decisions for a certain data, this embodiment makes a judgment with reference to the compressibility of the data arranged in front of the data. If the data arranged in front of the data is compressible, it is considered that the current data is also compressible. At the same time, the present application further judges the actual compression rate of the current data. When the actual compression rate of the current data is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification, so as to determine the compressibility of the data arranged behind the current data based on the updated identification information. It can be seen that this embodiment can refer to the relationship between the compressibility of adjacent data, and directly predict the compressibility of the data without calculation, completely avoiding the calculation overhead of the prediction. And after each actual compression is performed on the data, the size relationship between the actual compression rate and the threshold is compared to update the data compressibility mark in real time to avoid the compression mark being outdated and unable to represent the actual data compressibility. In this way, the compression rate prediction overhead can be minimized without affecting the accuracy of the compression rate prediction.
[0069] Based on the above embodiments, it can be known that if the compression rate of the current data block can reach a certain predefined compression rate threshold (i.e., a preset threshold), then the compression rate of the data blocks adjacent to its compression operation time is highly likely to reach the predefined compression rate threshold. Conversely, if the compression rate of the current data block cannot reach a certain predefined compression rate threshold, then the compression rate of the data blocks adjacent to its compression operation time is highly likely to also not reach the predefined compression rate threshold. Based on this, it can be considered that the compressibility of data has locality in the time of compression operation, that is, the compressibility of data has spatial locality of storage location and temporal locality of read and write operations. Therefore, data compression decisions can be made based on the storage location arrangement order or compression operation time arrangement order between different data without predicting the compression rate of the data.
[0070] In order to improve the prediction accuracy, this embodiment further combines the actual compression rate and information entropy of the data to determine the compressibility of the data. Figure 2 , this embodiment provides a data compression decision solution, including:
[0071] Step 1: Set the compression flag and define the compression ratio threshold.
[0072] Step 2: For each data block, first check the compression mark. If the compression mark is set, it is considered that the previous data block of the data block is compressible data. Based on the temporal and spatial locality of data compressibility, directly compress the current data block, and calculate the actual compression rate of the current data block and make a judgment. If the compression mark is not set, it is considered that the previous data block of the data block is incompressible data. After calculating the information entropy value of the current data block, predict the compression rate estimate based on the information entropy value and the fitted linear relationship between the information entropy and the compression rate, and make a judgment.
[0073] Step 3a: After calculating the actual compression rate of the current data block, check whether the actual compression rate is less than the compression rate threshold. If the actual compression rate is less than the compression rate threshold, it is considered that the actual compression rate meets the compression mark setting requirements, and the compression mark is set to indicate that the next data block of the current data block is compressible; otherwise, it is considered that the actual compression rate does not meet the requirements, and the compression mark is canceled to indicate that the next data block of the current data block is incompressible, and then a compression decision is made for the next data block based on whether there is a compression mark this time.
[0074] Step 3b: After calculating the estimated compression rate of the current data block, if the calculated estimated compression rate is less than the compression rate threshold, the process of compressing the current data block, calculating the actual compression rate of the current data block and making a judgment is started; otherwise, the current data block is considered incompressible, the compression mark is canceled, and then the next data block is processed.
[0075] In this embodiment, the estimated compression ratio and the actual compression ratio are compared with the same compression ratio threshold value. Of course, if necessary, the estimated compression ratio and the actual compression ratio can also be compared with different threshold values.
[0076] The compression marker is used to reflect that the compressibility of adjacent data is similar, which reduces the number of compression rate predictions and the corresponding prediction calculation overhead without affecting the prediction accuracy.
[0077] It should be noted that the relationship between data information entropy and compression rate is obtained by curve fitting the information entropy and compression rate of the data. After obtaining the relationship, the information entropy of the data can be substituted into the relationship to quickly calculate the estimated value of the compression rate. Among them, the information entropy of a data is used to represent the randomness of the data, and the randomness of the data is strongly related to the compressibility of the data. Specifically, the stronger the randomness of the data, the lower the compressibility. Therefore, the information entropy of the data can represent the compressibility of the data, so the information entropy of the data is generally used to make a qualitative judgment on whether the data is compressed, but the information entropy of the data cannot quantitatively estimate the compression rate of the data. However, with the help of the mathematical relationship between the information entropy of the data and the compression rate, the compression rate of the data can be quantitatively estimated through the information entropy of the data.
[0078] The following is an introduction using Shannon entropy as an example. Figure 3 , the Shannon entropy and compression rate of the data exist Figure 3 The relationship shown is that the larger the Shannon entropy of the data, the more incompressible the data is and the higher the compression rate value is. Figure 3 The horizontal axis value of each black point in the data represents its information entropy, also known as Shannon entropy, and the vertical axis value represents its compression rate. Based on the horizontal and vertical axis values of these black points, a curve can be fitted. The expression of the curve is the mathematical relationship between the data information entropy and the compression rate.
[0079] Through curve fitting, we can observe that the compression rate of data is approximately exponentially related to the Shannon entropy of data. Therefore, we can also deduce that the Shannon entropy of data and the logarithm of the compression rate of data are nearly linearly related by using mathematical properties. Next, we use the least squares polynomial to establish the linear relationship between the Shannon entropy of data and the logarithm of the data compression rate.
[0080] See also Figure 4 , divide a sample data into multiple data blocks of equal size, calculate the Shannon entropy of each data block, obtain the Shannon entropy array EL, and calculate the actual compression rate of each data block to obtain the compression rate array CRL, then perform least squares polynomial fitting on each element in the Shannon entropy array EL and each element in the compression rate array CRL, and calculate the polynomial parameters a and b. Then take the compression rate as the unknown quantity to be solved and the Shannon entropy as the variable to construct the corresponding mathematical formula R=e a*E+b -e b . Where R represents the compression ratio to be solved, a and b are polynomial parameters, E represents Shannon entropy, and e is the natural logarithm.
[0081] As a specific implementation, when calculating the polynomial parameters a and b through the least squares polynomial, the fitting function polyfit in the numpy library of python can be used, that is: a, b = polyfit (EL, log (CRL), 1). Among them, polyfit represents the linear fitting function based on the least squares polynomial fitting, EL is the Shannon entropy array, and CRL is the compression ratio array. Of course, in the specific implementation, the fitting function of other programming languages or other fitting methods can also be used for solution.
[0082] It can be seen that with the help of the linear relationship between the Shannon entropy of the data and the compression rate, the compression rate of the data can be quantitatively estimated through the Shannon entropy of the data, and the estimated compression rate thus predicted is more accurate.
[0083] In another example, this embodiment further combines the actual compression rate of the data and the sampling compression rate estimation method to accurately determine the compressibility of the data. Figure 5, this embodiment provides another data compression decision solution, including:
[0084] Step 1: Set the compression flag and define the compression ratio threshold.
[0085] Step 2: For each data block, first check the compression mark. If the compression mark is set, it is considered that the previous data block of the data block is compressible data. Based on the temporal and spatial locality of data compressibility, directly compress the current data block, and calculate the actual compression rate of the current data block and make a judgment. If the compression mark is not set, it is considered that the previous data block of the data block is incompressible data, and then calculate the sample compression rate of the current data block and make a judgment.
[0086] Step 3a: After calculating the actual compression rate of the current data block, check whether the actual compression rate is less than the compression rate threshold. If the actual compression rate is less than the compression rate threshold, it is considered that the actual compression rate meets the compression mark setting requirements, and the compression mark is set to indicate that the next data block of the current data block is compressible; otherwise, it is considered that the actual compression rate does not meet the requirements, and the compression mark is canceled to indicate that the next data block of the current data block is incompressible, and then a compression decision is made for the next data block based on whether there is a compression mark this time.
[0087] Step 3b: After calculating the sample compression rate of the current data block, if the calculated sample compression rate is less than the compression rate threshold, enter the process of compressing the current data block, calculate the actual compression rate of the current data block and make a judgment; otherwise, the current data block is considered incompressible, the compression mark is canceled, and then the next data block is processed.
[0088] The compression mark is used to reflect that the compressibility of adjacent data is similar, which can reduce the number and cost of compression rate prediction through sampling without affecting the prediction accuracy.
[0089] This application takes into account the fact that adjacent data has similar compressibility and sets corresponding compression marks. In theory, the compressibility of data can be effectively predicted based on the order of data without calculation. At the same time, this application combines the information entropy compression rate prediction method, the sampling compression rate prediction method, the mathematical expression compression rate prediction method, and the actual compression rate judgment, thereby improving the accuracy of data compression decision-making and reducing the calculation overhead of prediction.
[0090] A compression decision device provided in an embodiment of the present application is introduced below. The compression decision device described below and the compression decision method described above can be referenced to each other.
[0091] See also Figure 6 As shown, the embodiment of the present application discloses a compression decision device, including:
[0092] The acquisition module 601 is used for acquiring identification information for identifying whether the previous data of any target data is compressible if a compression decision is made for the target data; wherein the previous data and the target data are arranged in the order of compression operation time;
[0093] A decision module 602 is used to determine that the compression decision of the target data is compressible if the identification information is a compressible identification;
[0094] The updating module 603 is used to calculate the actual value of the compression ratio of the target data; if the actual value of the compression ratio is less than a preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification.
[0095] In a specific embodiment, it also includes:
[0096] An estimation module, used for calculating an estimated value of the compression ratio of the target data if the identification information is an incompressible identification;
[0097] The execution module is used to determine that the compression decision of the target data is compressible if the estimated value of the compression ratio is less than a preset threshold, and execute the steps in the calculation module and the decision module.
[0098] In a specific implementation, the estimation module is specifically used for:
[0099] Extracting sample data from target data, calculating an actual value of the compression rate of the sample data, and using the actual value of the compression rate of the sample data as an estimated value of the compression rate;
[0100] and / or
[0101] The information entropy of the target data is calculated, and the compression rate estimate is calculated based on the information entropy and a preset formula; the preset formula is obtained by fitting the relationship between the information entropy of the data and the actual compression rate.
[0102] In a specific implementation, the process of constructing the preset formula includes: obtaining multiple data blocks arranged in order; calculating the actual compression rate of each data block and the information entropy of each data block; fitting the actual compression rate of each data block and the information entropy of each data block to obtain a fitting relationship.
[0103] In a specific implementation, the actual compression rate of each data block and the information entropy of each data block are fitted, including: constructing the actual compression rate of each data block into a first array according to the arrangement order of each data block; constructing the information entropy of each data block into a second array according to the arrangement order of each data block; based on the first array and the second array, fitting the actual compression rate of each data block and the information entropy of each data block.
[0104] In a specific embodiment, it also includes:
[0105] The mean value calculation module is used to take the mean value of the at least two compression rate estimation values as the compression rate estimation value of the target data if at least two compression rate estimation values are obtained.
[0106] In a specific embodiment, it also includes:
[0107] The determination module is used to determine that the compression decision of the target data is incompressible if the estimated value of the compression ratio is not less than a preset threshold, and update the identification information to an incompressible identification.
[0108] In a specific embodiment, it also includes:
[0109] The default module is used to set the default identification information to a compressible identification if there is no previous data.
[0110] In a specific embodiment, it also includes:
[0111] A subsequent decision module is used to make a compression decision on the subsequent data of the target data according to the updated identification information; wherein the previous data, the target data and the subsequent data are arranged in the order of compression operation time.
[0112] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0113] It can be seen that this embodiment provides a compression decision device that can refer to the relationship between the compressibility of adjacent data and directly predict the compressibility of data without calculation, thereby completely avoiding the calculation overhead of the prediction.
[0114] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and the compression decision method and device described above can refer to each other.
[0115] See also Figure 7 As shown, the embodiment of the present application discloses an electronic device, including:
[0116] Memory 701, used for storing computer programs;
[0117] The processor 702 is used to execute the computer program to implement the method disclosed in any of the above embodiments.
[0118] A readable storage medium provided in an embodiment of the present application is introduced below. The readable storage medium described below and the compression decision method, device and apparatus described above can be referenced to each other.
[0119] A readable storage medium is used to store a computer program, wherein the computer program, when executed by a processor, implements the compression decision method disclosed in the above embodiment. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiment, and will not be repeated here.
[0120] The "first", "second", "third", "fourth", etc. (if any) referred to in this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods or devices.
[0121] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0122] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0123] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.
[0124] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A compression decision method, characterized in that: include: If a compression decision is made for any target data, identification information for identifying whether the previous data of the target data is compressible is obtained; wherein the previous data and the target data are arranged in the order of compression operation time, their storage addresses have spatial continuity, and their read and write operation times have temporal continuity; If the identification information is a compressible identification, determining that the compression decision of the target data is compressible, and calculating an actual value of the compression ratio of the target data; If the actual value of the compression ratio is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification; If the identification information is an incompressible identification, calculating an estimated value of the compression ratio of the target data; If the estimated compression ratio value is less than the preset threshold, the compression decision of the target data is determined to be compressible, and the step of calculating the actual compression ratio value of the target data is performed; if the actual compression ratio value is less than the preset threshold, the identification information is retained as a compressible identification; otherwise, the identification information is updated to an incompressible identification; a compression decision is made on the subsequent data of the target data according to the updated identification information; wherein the previous data, the target data and the subsequent data are arranged in the order of compression operation time; The step of calculating the estimated compression ratio of the target data includes: Calculating the information entropy of the target data, and calculating the compression rate estimation value based on the information entropy and a preset formula; the preset formula is obtained by fitting the relationship between the information entropy of the data and the actual compression rate; The process of constructing the preset formula includes: Acquire multiple data blocks arranged in sequence; the data blocks are: data blocks of equal size constituting the same file or data blocks of different sizes belonging to different files; Calculate the actual compression ratio of each data block and the information entropy of each data block; Fitting the actual compression rate of each data block and the information entropy of each data block to obtain the preset formula; The step of fitting the actual compression ratio of each data block and the information entropy of each data block includes: Constructing the actual compression ratio of each data block into a first array according to the arrangement order of each data block; Constructing the information entropy of each data block into a second array according to the arrangement order of each data block; Based on the first array and the second array, a polynomial is used to fit the actual compression rate of each data block and the information entropy of each data block.
2. The method according to claim 1, characterized in that The calculating the estimated value of the compression ratio of the target data comprises: Sample data is extracted from the target data, an actual value of the compression ratio of the sample data is calculated, and the actual value of the compression ratio of the sample data is used as the estimated value of the compression ratio.
3. The method according to claim 1, characterized in that Also includes: If at least two compression ratio estimation values are obtained, an average value of the at least two compression ratio estimation values is taken as the compression ratio estimation value of the target data.
4. The method according to claim 1, characterized in that Also includes: If the compression ratio estimation value is not less than the preset threshold, the compression decision of the target data is determined to be incompressible, and the identification information is updated to an incompressible identification.
5. The method according to any one of claims 1 to 4, characterized in that: Also includes: If the previous data does not exist, the identification information is assumed to be a compressible identification.
6. A compression decision device, characterized in that: include: An acquisition module, used for acquiring identification information for identifying whether the previous data of any target data is compressible if a compression decision is made for the target data; wherein the previous data and the target data are arranged in the order of compression operation time, their storage addresses have spatial continuity, and their read and write operation times have temporal continuity; A decision module, configured to determine that the compression decision of the target data is compressible if the identification information is a compressible identification; and to make a compression decision on the subsequent data of the target data according to the updated identification information; wherein the previous data, the target data and the subsequent data are arranged in the order of compression operation time; An updating module, used for calculating an actual value of the compression ratio of the target data; if the actual value of the compression ratio is less than a preset threshold, retaining the identification information as a compressible identification; otherwise, updating the identification information to an incompressible identification; an estimation module, configured to calculate an estimated value of the compression ratio of the target data if the identification information is an incompressible identification; an execution module, configured to determine that the compression decision of the target data is compressible if the estimated compression ratio value is less than the preset threshold, and execute the step of calculating the actual compression ratio value of the target data; if the actual compression ratio value is less than the preset threshold, retain the identification information as a compressible identification; otherwise, update the identification information to an incompressible identification; The estimation module is specifically used for: Calculating the information entropy of the target data, and calculating the compression rate estimation value based on the information entropy and a preset formula; the preset formula is obtained by fitting the relationship between the information entropy of the data and the actual compression rate; The process of constructing the preset formula includes: Acquire multiple data blocks arranged in sequence; the data blocks are: data blocks of equal size constituting the same file or data blocks of different sizes belonging to different files; Calculate the actual compression ratio of each data block and the information entropy of each data block; Fitting the actual compression rate of each data block and the information entropy of each data block to obtain the preset formula; The step of fitting the actual compression ratio of each data block and the information entropy of each data block includes: Constructing the actual compression ratio of each data block into a first array according to the arrangement order of each data block; Constructing the information entropy of each data block into a second array according to the arrangement order of each data block; Based on the first array and the second array, a polynomial is used to fit the actual compression rate of each data block and the information entropy of each data block.
7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 5.
8. A readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
