Cloud edge end data two-stage compression method and system

Abstract

The invention discloses a cloud side end data two-stage compression method and system, and belongs to the technical field of data compression. The problem that a traditional compression method is low in compression rate is solved. According to the method, firstly, the requirement of a task for data integrity is analyzed, the data integrity requirement serves as a constraint, the maximum compression ratio and the minimum loss ratio serve as targets, the proportion of lossy compression and lossless compression is calculated, then a compression method is selected according to the proportion to compress data, and therefore elastic compression is achieved. On the basis of primary compression, whether secondary compression is carried out or not is judged according to a compression result, if secondary compression needs to be carried out, a secondary compression method is selected according to data distribution characteristics after primary compression, secondary compression is carried out on the primary compression result, a higher compression ratio can be obtained, and the higher compression ratio is achieved. The method can be applied to the technical field of data compression.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and specifically relates to a two-stage compression method and system for cloud-edge data. Background Art

[0002] To achieve efficient collaborative data management between cloud, edge and end, collaborative storage must be the foundation. The sources of cloud, edge and end data are diverse and huge, including both input data and real-time data collected by various sensors. Traditional methods are difficult to meet the requirements of collaborative storage for massive multi-mode heterogeneous data in cloud, edge and end collaborative scenarios.

[0003] The data size on the edge side may be very large and there may be a lot of redundancy, which will burden transmission and storage. It is necessary to consider the characteristics of cloud-edge-end data applications for compression and storage. However, the compression rate of traditional methods is generally not high, and the low compression rate still leads to great pressure on transmission and storage.

[0004] Therefore, in response to the above shortcomings, a new intelligent compression method for cloud-edge data is proposed, which is very necessary to achieve collaborative intelligent high-compression rate compression of cloud-edge data. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of low compression rate of traditional compression methods, and to propose a two-level compression method and system for cloud-edge data.

[0006] The technical solution adopted by the present invention to solve the above technical problems is:

[0007] According to one aspect of the present invention, a two-stage compression method for cloud-edge-end data is provided, and the method specifically comprises the following steps:

[0008] Step 1: determine the ratio of lossy compressed data to lossless compressed data, and then select a compression method according to the ratio of lossy compressed data to lossless compressed data. After compressing the data using the selected compression method, the compressed data is stored in a tuple storage mode to complete the first-level data compression.

[0009] Step 2: Calculate the redundancy of the first-level data compression result;

[0010] If the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, execute step three;

[0011] Otherwise, the entire data compression process ends;

[0012] Step 3: Perform secondary compression on the primary data compression result, and end the entire compression process after the secondary compression is completed.

[0013] According to another aspect of the present invention, a two-stage compression system for cloud-edge data is provided, the system comprising a data compression ratio calculation module, a primary compression module and a secondary compression module; wherein:

[0014] The data compression ratio calculation module is used to calculate the ratio of lossy compressed data to lossless compressed data;

[0015] The primary compression module is used to select a compression method according to the ratio of lossy compressed data to lossless compressed data, and use the selected compression method to compress the lossy compressed data and the lossless compressed data respectively, and use a tuple storage method to store the compressed data;

[0016] Then determine whether to start the second-level compression, that is, if the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, then start the second-level compression; otherwise, do not start the second-level compression;

[0017] The secondary compression module is used to perform secondary compression on the primary compression result.

[0018] The beneficial effects of the present invention are:

[0019] The present invention first analyzes the requirements of the task for data integrity, takes the data integrity requirements as constraints, and aims to maximize the compression ratio and minimize the loss rate, calculates the ratio of lossy compression and lossless compression, and then selects a compression method to compress the data according to the ratio, thereby achieving elastic compression. On the basis of primary compression, it is determined whether to perform secondary compression according to the compression result. If secondary compression is required, the secondary compression method is selected according to the data distribution characteristics after primary compression, and the primary compression result is subjected to secondary compression, so that a higher compression ratio can be obtained, achieving a higher compression rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a two-stage compression method of cloud-edge-end data of the present invention;

[0021] Figure 2 This is an architecture diagram of a two-stage compression system for cloud-edge data of the present invention. DETAILED DESCRIPTION

[0022] Specific implementation method 1: Combination Figure 1 The present embodiment describes a two-stage cloud-edge-end data compression method, the method specifically comprising the following steps:

[0023] Step 1: Determine the ratio of lossy compressed data to lossless compressed data, and then select a compression method according to the ratio of lossy compressed data to lossless compressed data. After compressing the data using the selected compression method, the compressed data is stored in a tuple storage mode on the memory (the row and column storage is automatically converted into a tuple storage mode, which can realize compression oriented to the data storage structure), and complete the first-level data compression;

[0024] Step 2: Calculate the redundancy of the first-level data compression result;

[0025] If the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, execute step three;

[0026] The threshold in the present invention can be determined according to the memory and data transmission rate that the system can provide, and an acceptable threshold is set on the premise of satisfying the actual memory and data transmission rate;

[0027] Otherwise, the entire data compression process ends;

[0028] Step 3: Perform secondary compression on the primary data compression result, and end the entire compression process after the secondary compression is completed.

[0029] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that, in step 1, the specific method for determining the ratio of lossy compressed data to lossless compressed data is:

[0030] Step 11: Initialize the proportion of lossy compressed data to p%, and the proportion of lossless compressed data to (100-p)%; where p is an integer, which can be 50 in general;

[0031] Step 12: Select a lossy compression method according to the ratio of lossy compressed data, and use the selected lossy compression method to compress the data that needs lossy compression; select a lossless compression method according to the ratio of lossless compressed data, and use the selected lossless compression method to compress the data that needs lossless compression;

[0032] Calculate the objective function value F under the current compression ratio according to the compression time T, the compressed data size C, and the data loss rate L;

[0033] Step 13: adjust the proportion of lossy compressed data to (p-1)%, the proportion of lossless compressed data to (101-p)%, and then use the method of step 12 to calculate the objective function value F1 after adjusting the compression ratio; adjust the proportion of lossy compressed data to (p+1)%, the proportion of lossless compressed data to (99-p)%, and then use the method of step 12 to calculate the objective function value F2 after adjusting the compression ratio;

[0034] Compare the sizes of F, F1 and F2;

[0035] If F1<F<F2, then execute step 14;

[0036] If F2<F<F1, then execute step 17;

[0037] If F2≥F and F≤F1, the proportion of lossy compressed data is determined to be p%, and the proportion of lossless compressed data is determined to be (100-p)%;

[0038] Step 14: Set the number of iterations a=1;

[0039] Step 15: adjust the proportion of lossy compressed data to (p-1-a)%, and the proportion of lossless compressed data to (101-p+a)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration;

[0040] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0041] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 16;

[0042] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0043] Step 16: Let a=a+1, and return to execute step 15;

[0044] Step 17: Set the number of iterations a=1;

[0045] Step 18: Adjust the proportion of lossy compressed data to (p+1+a)%, and the proportion of lossless compressed data to (99-pa)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration;

[0046] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0047] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 19;

[0048] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0049] Step 19: Let a=a+1, and return to execute step 18.

[0050] The other steps and parameters are the same as those in the first embodiment.

[0051] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the objective function value is Min F (C, T, L);

[0052] F(C,T,L)=w1*f1(C)+w2*f2(T)+w3*f3(L)

[0053] Among them, F(C, T, L) is the objective function, w1, w2 and w3 are weight coefficients, w1+w2+w3=1, f1(C)=C, f2(T)=T, f3(L)=L.

[0054] The other steps and parameters are the same as those in the first or second embodiment.

[0055] The present invention takes the requirement of computing task on data integrity as constraint, takes minimizing storage scale and task execution efficiency as optimization goal, designs a combinatorial optimization algorithm to calculate the ratio of lossy compressed data to lossless compressed data, that is, determines the objective function value under each ratio respectively, and then selects the appropriate compression ratio by comparing the objective function values.

[0056] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the compression method is selected according to the ratio of lossy compressed data to lossless compressed data, specifically:

[0057] For lossless compressed data:

[0058] If the proportion of lossless compressed data is (70%, 100%], the selected lossless compression method is RLE, Delta of Delta, XOR, Offset, or Dictionary;

[0059] If the proportion of lossless compressed data is (40%, 70%], the selected lossless compression method is Delta of Delta or Dictionary;

[0060] If the proportion of lossless compressed data is [0%, 40%], the selected lossless compression method is Dictionary;

[0061] For any proportion of lossy compressed data, the selected lossy compression method is Trailing-zero.

[0062] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0063] The first-level compression adopts a lightweight compression encoding method to improve the compression efficiency while balancing the compression rate and improving the overall throughput of the system. The data encoding methods are shown in Table 1:

[0064] Table 1

[0065]

[0066]

[0067] Specific implementation method 5: This implementation method is different from the specific implementation methods 1 to 4 in that the primary data compression result is subjected to secondary compression, and the specific process is as follows:

[0068] Step 31: Encode the first-level lossy compression result to obtain an encoding result corresponding to the lossy compression result, and then encode the first-level lossless compression result to obtain an encoding result corresponding to the lossless compression result;

[0069] Step 32: The encoding result corresponding to the lossy compression result and the encoding result corresponding to the lossless compression result are respectively passed through a deep learning network, and the deep learning network is used to extract the features of the first-level lossy compression result and the features of the first-level lossless compression result;

[0070] Step 33: Select a compression method based on the characteristics of the first-level lossy compression result, and then use the selected compression method to perform secondary compression on the first-level lossy compression result; select a compression method based on the characteristics of the first-level lossless compression result, and then use the selected compression method to perform secondary compression on the first-level lossless compression result.

[0071] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0072] The elastic compression method includes several compression schemes, as shown in Table 2. The most suitable compression scheme is selected as the compression method through the output of the deep learning network, thereby achieving secondary efficient compression of the data.

[0073] Table 2

[0074]

[0075] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the deep learning network includes a Bi-LSTM module, a fully connected layer and a Softmax layer; and the working process of the deep learning network is:

[0076] Take the encoding result corresponding to the lossy compression result as an example

[0077] The encoding result corresponding to the lossy compression result is used as the input of the Bi-LSTM module, and the output of the Bi-LSTM module is used as the input of the fully connected layer, and the output of the fully connected layer is used as the input of the Softmax layer, and the features of the lossy compression result are output through the Softmax layer.

[0078] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0079] The Bi-LSTM module contains forward and backward layers to capture past and future information. The Dense layer (fully connected layer) maps the learned features to the sample space. The Softmax layer is used to convert the model output results into probabilities to find the maximum probability item. The deep learning network uses massive data for vectorized feature extraction training. Through deep learning, each data feature can be effectively extracted, and the data can be compressed at the second level based on the extracted data features.

[0080] Specific implementation method seven: Combination Figure 2 This embodiment describes a two-stage cloud-edge data compression system, the system comprising a data compression ratio calculation module, a primary compression module and a secondary compression module; wherein:

[0081] The data compression ratio calculation module is used to calculate the ratio of lossy compressed data to lossless compressed data;

[0082] The primary compression module is used to select a compression method according to the ratio of lossy compressed data to lossless compressed data, and use the selected compression method to compress the lossy compressed data and the lossless compressed data respectively, and use a tuple storage method to store the compressed data in the memory;

[0083] Then determine whether to start secondary compression, that is

[0084] If the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, the second-level compression is started; otherwise, the second-level compression is not started;

[0085] The secondary compression module is used to perform secondary compression on the primary compression result.

[0086] Specific implementation eight: This implementation is different from specific implementation seven in that the working process of the data compression ratio calculation module is as follows:

[0087] Step 1: Initialize the proportion of lossy compressed data to p%, and the proportion of lossless compressed data to (100-p)%; where p is an integer, which can be 50 in general;

[0088] Step 2: Select a lossy compression method according to the ratio of lossy compressed data, and use the selected lossy compression method to compress the data that needs lossy compression; select a lossless compression method according to the ratio of lossless compressed data, and use the selected lossless compression method to compress the data that needs lossless compression;

[0089] For lossless compressed data:

[0090] If the proportion of lossless compressed data is (70%, 100%], the selected lossless compression method is RLE, Delta of Delta, XOR, Offset, or Dictionary;

[0091] If the proportion of lossless compressed data is (40%, 70%], the selected lossless compression method is Delta of Delta or Dictionary;

[0092] If the proportion of lossless compressed data is [0%, 40%], the selected lossless compression method is Dictionary;

[0093] For lossy compressed data of any proportion, the selected lossy compression method is Trailing-zero;

[0094] Calculate the objective function value F under the current compression ratio according to the compression time T, the compressed data size C, and the data loss rate L;

[0095] Step 3: Adjust the proportion of lossy compressed data to (p-1)%, the proportion of lossless compressed data to (101-p)%, and then use the method of step 2 to calculate the objective function value F1 after adjusting the compression ratio; adjust the proportion of lossy compressed data to (p+1)%, the proportion of lossless compressed data to (99-p)%, and then use the method of step 2 to calculate the objective function value F2 after adjusting the compression ratio;

[0096] Compare the sizes of F, F1 and F2;

[0097] If F1<F<F2, execute step 4;

[0098] If F2<F<F1, execute step 7;

[0099] If F2≥F and F≤F1, the proportion of lossy compressed data is determined to be p%, and the proportion of lossless compressed data is determined to be (100-p)%;

[0100] Step 4: Set the number of iterations a=1;

[0101] Step 5: adjust the proportion of lossy compressed data to (p-1-a)%, and the proportion of lossless compressed data to (101-p+a)%, and then use the method of step 2 to calculate the objective function value obtained in the current iteration;

[0102] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0103] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 6;

[0104] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0105] Step 6: Let a=a+1, and return to step 5;

[0106] Step 7: Set the number of iterations a=1;

[0107] Step 8: Adjust the proportion of lossy compressed data to (p+1+a)%, and the proportion of lossless compressed data to (99-pa)%, and then use the method of step 2 to calculate the objective function value obtained in the current iteration;

[0108] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0109] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step nine;

[0110] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0111] Step 9: Let a=a+1, and return to execute step 8.

[0112] The other steps and parameters are the same as those in the seventh embodiment.

[0113] Specific implementation method 9: This implementation method is different from specific implementation method 7 or 8 in that the objective function value is Min F (C, T, L);

[0114] F(C,T,L)=w1*f1(C)+w2*f2(T)+w3*f3(L)

[0115] Among them, F(C, T, L) is the objective function, w1, w2 and w3 are weight coefficients, w1+w2+w3=1, f1(C)=C, f2(T)=T, f3(L)=L.

[0116] The other steps and parameters are the same as those in the seventh or eighth specific implementation mode.

[0117] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 7 to 9 in that the specific process of the secondary compression module is as follows:

[0118] Step 1, encode the first-level lossy compression result to obtain an encoding result corresponding to the lossy compression result, and then encode the first-level lossless compression result to obtain an encoding result corresponding to the lossless compression result;

[0119] Step 2: The encoding result corresponding to the lossy compression result and the encoding result corresponding to the lossless compression result are respectively passed through a deep learning network, and the deep learning network is used to extract the features of the first-level lossy compression result and the features of the first-level lossless compression result;

[0120] The deep learning network includes a Bi-LSTM module, a fully connected layer and a Softmax layer; and the working process of the deep learning network is:

[0121] Take the encoding result corresponding to the lossy compression result as an example

[0122] The encoding result corresponding to the lossy compression result is used as the input of the Bi-LSTM module, and the output of the Bi-LSTM module is used as the input of the fully connected layer, and the output of the fully connected layer is used as the input of the Softmax layer, and the features of the lossy compression result are output through the Softmax layer;

[0123] Step 3: Select a compression method based on the characteristics of the first-level lossy compression result, and then use the selected compression method to perform secondary compression on the first-level lossy compression result; select a compression method based on the characteristics of the first-level lossless compression result, and then use the selected compression method to perform secondary compression on the first-level lossless compression result.

[0124] The other steps and parameters are the same as those in Specific Embodiments 7 to 9.

[0125] Example

[0126] In the field of Internet of Things, sensors and devices collect a large amount of data such as machine status, production line operation, temperature, humidity and vibration. These data are large in volume, frequently updated and have time series characteristics, which puts extremely high demands on storage and scheduling. For example: machine status data (each device reports status information once a minute, including temperature, vibration value, energy consumption, etc.); environmental monitoring data (sensors record environmental temperature, humidity and air pressure information every 5 seconds). These huge amounts of data are stored in the database, which puts higher requirements on the data compression rate. The present invention proposes a data compression method applicable to both machine status data and environmental monitoring data.

[0127] Taking machine status data as an example, the data compression method of the present invention is described as follows:

[0128] Step 1: Determine the part of the machine status data that needs lossy compression and the part that needs lossless compression, that is, determine the ratio of lossy compressed data to lossless compressed data, specifically:

[0129] Step 11: Initialize the proportion of lossy compressed data to p%, and the proportion of lossless compressed data to (100-p)%; wherein p is an integer, which can be generally taken as 50, that is, except for the part that needs lossy compression in the machine state data, the rest are the parts that need lossless compression;

[0130] Step 12: Select a lossy compression method according to the ratio of lossy compressed data, and use the selected lossy compression method to compress the data that needs lossy compression; select a lossless compression method according to the ratio of lossless compressed data, and use the selected lossless compression method to compress the data that needs lossless compression;

[0131] For lossless compressed data:

[0132] If the proportion of lossless compressed data is (70%, 100%], the selected lossless compression method is RLE, Delta of Delta, XOR, Offset, or Dictionary;

[0133] If the proportion of lossless compressed data is (40%, 70%], the selected lossless compression method is Delta of Delta or Dictionary;

[0134] If the proportion of lossless compressed data is [0%, 40%], the selected lossless compression method is Dictionary;

[0135] For any proportion of lossy compressed data, the selected lossy compression method is Trailing-zero.

[0136] Calculate the objective function value F under the current compression ratio according to the compression time T, the compressed data size C, and the data loss rate L;

[0137] Step 13: adjust the proportion of lossy compressed data to (p-1)%, the proportion of lossless compressed data to (101-p)%, and then use the method of step 12 to calculate the objective function value F1 after adjusting the compression ratio; adjust the proportion of lossy compressed data to (p+1)%, the proportion of lossless compressed data to (99-p)%, and then use the method of step 12 to calculate the objective function value F2 after adjusting the compression ratio;

[0138] Compare the sizes of F, F1 and F2;

[0139] If F1<F<F2, then execute step 14;

[0140] If F2<F<F1, then execute step 17;

[0141] If F2≥F and F≤F1, the proportion of lossy compressed data is determined to be p%, and the proportion of lossless compressed data is determined to be (100-p)%;

[0142] Step 14: Set the number of iterations a=1;

[0143] Step 15: adjust the proportion of lossy compressed data to (p-1-a)%, and the proportion of lossless compressed data to (101-p+a)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration;

[0144] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0145] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 16;

[0146] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0147] Step 16: Let a=a+1, and return to execute step 15;

[0148] Step 17: Set the number of iterations a=1;

[0149] Step 18: Adjust the proportion of lossy compressed data to (p+1+a)%, and the proportion of lossless compressed data to (99-pa)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration;

[0150] Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration;

[0151] If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 19;

[0152] If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios;

[0153] Step 19: Let a=a+1, and return to execute step 18.

[0154] Then, a compression method is selected according to the ratio of lossy compressed data to lossless compressed data. After the data is compressed using the selected compression method, the compressed data is stored in a tuple storage mode on the memory (the row and column storage is automatically converted into a tuple storage mode, which can realize compression for the data storage structure), completing the first-level data compression;

[0155] Step 2: Calculate the redundancy of the first-level data compression result;

[0156] If the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, execute step three;

[0157] The threshold in the present invention can be determined according to the memory and data transmission rate that the system can provide, and an acceptable threshold is set on the premise of satisfying the actual memory and data transmission rate;

[0158] Otherwise, the entire data compression process ends;

[0159] Step 3: Perform secondary compression on the primary data compression result, specifically:

[0160] Step 31: Encode the first-level lossy compression result to obtain an encoding result corresponding to the lossy compression result, and then encode the first-level lossless compression result to obtain an encoding result corresponding to the lossless compression result;

[0161] Step 32: The encoding result corresponding to the lossy compression result and the encoding result corresponding to the lossless compression result are respectively passed through a deep learning network, and the deep learning network is used to extract the features of the first-level lossy compression result and the features of the first-level lossless compression result;

[0162] The deep learning network includes Bi-LSTM modules, fully connected layers and Softmax layers; and the working process of the deep learning network is:

[0163] Take the encoding result corresponding to the lossy compression result as an example

[0164] The encoding result corresponding to the lossy compression result is used as the input of the Bi-LSTM module, and the output of the Bi-LSTM module is used as the input of the fully connected layer, and the output of the fully connected layer is used as the input of the Softmax layer, and the features of the lossy compression result are output through the Softmax layer.

[0165] Step 33: Select a compression method based on the characteristics of the first-level lossy compression result, and then use the selected compression method to perform secondary compression on the first-level lossy compression result; select a compression method based on the characteristics of the first-level lossless compression result, and then use the selected compression method to perform secondary compression on the first-level lossless compression result to obtain the secondary compression result of the machine status data.

[0166] It should also be noted that the calculation method of the objective function value Min F(C, T, L) in the present invention is as follows:

[0167] F(C,T,L)=w1*f1(C)+w2*f2(T)+w3*f3(L)

[0168] Among them, F(C, T, L) is the objective function, w1, w2 and w3 are weight coefficients, w1+w2+u3=1, f1(C)=C, f2(T)=T, f3(L)=L.

[0169] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A two-level compression method for cloud-edge data, characterized in that: The method specifically comprises the following steps: Step 1: determine the ratio of lossy compressed data to lossless compressed data, and then select a compression method according to the ratio of lossy compressed data to lossless compressed data. After compressing the data using the selected compression method, the compressed data is stored in a tuple storage mode to complete the first-level data compression. Step 2: Calculate the redundancy of the first-level data compression result; If the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, execute step three; Otherwise, the entire data compression process ends; Step 3: Perform secondary compression on the primary data compression result, and end the entire compression process after the secondary compression is completed.

2. According to claim 1, a two-stage compression method for cloud-edge data is characterized in that: In step 1, the specific method for determining the ratio of lossy compressed data to lossless compressed data is: Step 11: Initialize the proportion of lossy compressed data to p%, and the proportion of lossless compressed data to (100-p)%; where p is an integer; Step 12: Select a lossy compression method according to the ratio of lossy compressed data, and use the selected lossy compression method to compress the data that needs lossy compression; select a lossless compression method according to the ratio of lossless compressed data, and use the selected lossless compression method to compress the data that needs lossless compression; Calculate the objective function value F under the current compression ratio according to the compression time T, the compressed data size C, and the data loss rate L; Step 13: adjust the proportion of lossy compressed data to (p-1)%, the proportion of lossless compressed data to (101-p)%, and then use the method of step 12 to calculate the objective function value F1 after adjusting the compression ratio; adjust the proportion of lossy compressed data to (p+1)%, the proportion of lossless compressed data to (99-p)%, and then use the method of step 12 to calculate the objective function value F2 after adjusting the compression ratio; Compare the sizes of F, F1 and F2; If F1<F<F2, then execute step 14; If F2<F<F1, then execute step 17; If F2≥F and F≤F1, the proportion of lossy compressed data is determined to be p%, and the proportion of lossless compressed data is determined to be (100-p)%; Step 14: Set the number of iterations a=1; Step 15: adjust the proportion of lossy compressed data to (p-1-a)%, and the proportion of lossless compressed data to (101-p+a)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration; Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration; If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 16; If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios; Step 16: Let a=a+1, and return to execute step 15; Step 17: Set the number of iterations a=1; Step 18: Adjust the proportion of lossy compressed data to (p+1+a)%, and the proportion of lossless compressed data to (99-pa)%, and then use the method of step 12 to calculate the objective function value obtained in the current iteration; Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration; If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 19; If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios; Step 19: Let a=a+1, and return to execute step 18.

3. According to claim 2, a two-stage compression method for cloud-edge data is characterized in that: The objective function value is Min F(C, T, L); F(C,T,L)=w1*f1(c)+w2*f2(T)+w3*f3(L) Among them, F(C, T, L) is the objective function, w1, w2 and w3 are weight coefficients, w1+w2+w3=1, f1(C)=C, f2(L)=T, f3(L)=L.

4. According to claim 3, a two-stage compression method for cloud-edge data is characterized in that: The compression method is selected according to the ratio of lossy compressed data to lossless compressed data, specifically: For lossless compressed data: If the proportion of lossless compressed data is (70%, 100%], the selected lossless compression method is RLE, Delta of Delta, XOR, Offset, or Dictionary; If the proportion of lossless compressed data is (40%, 70%], the selected lossless compression method is Delta of Delta or Dictionary; If the proportion of lossless compressed data is [0%, 40%], the selected lossless compression method is Dictionary; For any proportion of lossy compressed data, the selected lossy compression method is Trailing-zero.

5. According to claim 4, a two-stage compression method for cloud-edge data is characterized in that: The specific process of performing secondary compression on the primary data compression result is as follows: Step 31: Encode the first-level lossy compression result to obtain an encoding result corresponding to the lossy compression result, and then encode the first-level lossless compression result to obtain an encoding result corresponding to the lossless compression result; Step 32: The encoding result corresponding to the lossy compression result and the encoding result corresponding to the lossless compression result are respectively passed through a deep learning network, and the deep learning network is used to extract the features of the first-level lossy compression result and the features of the first-level lossless compression result; Step 33: selecting a compression method according to the characteristics of the primary lossy compression result, and then performing secondary compression on the primary lossy compression result using the selected compression method; A compression method is selected according to the characteristics of the primary lossless compression result, and then the primary lossless compression result is subjected to secondary compression using the selected compression method.

6. A two-stage cloud-edge-end data compression method according to claim 5, characterized in that: The deep learning network includes a Bi-LSTM module, a fully connected layer and a Softmax layer; and the working process of the deep learning network is: Take the encoding result corresponding to the lossy compression result as an example The encoding result corresponding to the lossy compression result is used as the input of the Bi-LSTM module, and the output of the Bi-LSTM module is used as the input of the fully connected layer, and the output of the fully connected layer is used as the input of the Softmax layer, and the features of the lossy compression result are output through the Softmax layer.

7. A two-stage compression system for cloud-edge data, characterized in that: The system includes a data compression ratio calculation module, a primary compression module and a secondary compression module; wherein: The data compression ratio calculation module is used to calculate the ratio of lossy compressed data to lossless compressed data; The primary compression module is used to select a compression method according to the ratio of lossy compressed data to lossless compressed data, and use the selected compression method to compress the lossy compressed data and the lossless compressed data respectively, and use a tuple storage method to store the compressed data; Then determine whether to start the second-level compression, that is, if the redundancy is greater than the threshold or the size of the first-level data compression result exceeds the specified threshold, then start the second-level compression; otherwise, do not start the second-level compression; The secondary compression module is used to perform secondary compression on the primary compression result.

8. A two-stage cloud-edge-end data compression system according to claim 7, characterized in that: The working process of the data compression ratio calculation module is as follows: Step 1: Initialize the proportion of lossy compressed data to p%, and the proportion of lossless compressed data to (100-p)%; where p is an integer; Step 2: Select a lossy compression method according to the ratio of lossy compressed data, and use the selected lossy compression method to compress the data that needs lossy compression; select a lossless compression method according to the ratio of lossless compressed data, and use the selected lossless compression method to compress the data that needs lossless compression; For lossless compressed data: If the proportion of lossless compressed data is (70%, 100%], the selected lossless compression method is RLE, Delta of Delta, XOR, Offset, or Dictionary; If the proportion of lossless compressed data is (40%, 70%], the selected lossless compression method is Delta of Delta or Dictionary; If the proportion of lossless compressed data is [0%, 40%], the selected lossless compression method is Dictionary; For lossy compressed data of any proportion, the selected lossy compression method is Trailing-zero; Calculate the objective function value F under the current compression ratio according to the compression time T, the compressed data size C, and the data loss rate L; Step 3: Adjust the proportion of lossy compressed data to (p-1)%, the proportion of lossless compressed data to (101-p)%, and then use the method of step 2 to calculate the objective function value F1 after adjusting the compression ratio; adjust the proportion of lossy compressed data to (p+1)%, the proportion of lossless compressed data to (99-p)%, and then use the method of step 2 to calculate the objective function value F2 after adjusting the compression ratio; Compare the sizes of F, F1 and F2; If F1<F<F2, execute step 4; If F2<F<F1, execute step 7; If F2≥F and F≤F1, the proportion of lossy compressed data is determined to be p%, and the proportion of lossless compressed data is determined to be (100-p)%; Step 4: Set the number of iterations a=1; Step 5: adjust the proportion of lossy compressed data to (p-1-a)%, and the proportion of lossless compressed data to (101-p+a)%, and then use the method of step 2 to calculate the objective function value obtained in the current iteration; Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration; If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step 6; If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios; Step 6: Let a=a+1, and return to step 5; Step 7: Set the number of iterations a=1; Step 8: Adjust the proportion of lossy compressed data to (p+1+a)%, and the proportion of lossless compressed data to (99-pa)%, and then use the method of step 2 to calculate the objective function value obtained in the current iteration; Then compare the objective function value obtained in the current iteration with the objective function value obtained in the previous iteration; If the objective function value obtained in the current iteration is less than the objective function value obtained in the previous iteration, execute step nine; If the objective function value obtained in the current iteration is greater than or equal to the objective function value obtained in the previous iteration, the lossy compressed data ratio and the lossless compressed data ratio in the previous iteration are the determined ratios; Step 9: Let a=a+1, and return to execute step 8.

9. A two-stage cloud-edge-end data compression system according to claim 8, characterized in that: The objective function value is Min F(C, T, L); F(C,T,L)=w1*f1(C)+w2*f2(T)+w3*f3(L) Among them, F(C, T, L) is the objective function, w1, w2 and w3 are weight coefficients, w1+w2+w3=1, f1(C)=C, f2(T)=T, f3(L)=L.

10. A two-stage cloud-edge-end data compression system according to claim 9, characterized in that: The specific process of the secondary compression module is as follows: Step 1, encoding the first-level lossy compression result to obtain an encoding result corresponding to the lossy compression result, and then encoding the first-level lossless compression result to obtain an encoding result corresponding to the lossless compression result; Step 2: The encoding result corresponding to the lossy compression result and the encoding result corresponding to the lossless compression result are respectively passed through a deep learning network, and the deep learning network is used to extract the features of the first-level lossy compression result and the features of the first-level lossless compression result; The deep learning network includes a Bi-LSTM module, a fully connected layer and a Softmax layer; and the working process of the deep learning network is: Take the encoding result corresponding to the lossy compression result as an example The encoding result corresponding to the lossy compression result is used as the input of the Bi-LSTM module, and the output of the Bi-LSTM module is used as the input of the fully connected layer, and the output of the fully connected layer is used as the input of the Softmax layer, and the features of the lossy compression result are output through the Softmax layer; Step 3, selecting a compression method according to the characteristics of the primary lossy compression result, and then performing secondary compression on the primary lossy compression result using the selected compression method; A compression method is selected according to the characteristics of the primary lossless compression result, and then the primary lossless compression result is subjected to secondary compression using the selected compression method.

Images