Efficient data compression encryption and decryption method under low bandwidth condition

Abstract

According to the efficient data compression encryption and decryption method under the low-bandwidth condition, aiming at the characteristics of different data types, a specific compression algorithm is adopted to improve the compression ratio and the flexibility, and different encryption levels are adopted according to the data sensitivity degree to ensure the safety and the resource utilization efficiency. Meanwhile, by optimizing the time of all links such as compression, encryption, transmission, decryption and decompression, balance of the overall time is achieved, and therefore the overall efficiency of the system and the user experience are improved. According to the invention, some problems existing in data compression, encryption and decryption under the narrow bandwidth condition at present are effectively solved.

Description

Technical Field

[0001] The present invention relates to the fields of data compression and encryption technology, communication technology and information security technology, and in particular to a high-efficiency data compression encryption and decryption method under low bandwidth conditions. Background Art

[0002] In modern communication networks, many practical application scenarios have bandwidth limitations, such as communications between ships and shores at sea, network connections in remote areas, IoT devices, wireless sensor networks, and satellite communications. The devices and network infrastructure in these scenarios usually cannot provide high bandwidth support, so efficient data compression and encryption and decryption methods are needed to improve transmission efficiency and security.

[0003] Under narrow bandwidth conditions, using a single compression algorithm may not achieve the expected results. Therefore, different compression algorithms need to be selected for different data to improve the compression rate and ensure data quality. At the same time, with the increase in network attacks and the improvement of data privacy awareness, the importance of data encryption technology has become increasingly prominent. Advanced encryption algorithms can ensure efficient and secure data transmission under narrow bandwidth conditions.

[0004] Many current compression and encryption technologies perform well in high-bandwidth, high-performance computing environments, but perform poorly in narrow-bandwidth, low-power application scenarios. Existing technologies often cannot take into account the needs of efficient compression and secure encryption at the same time, or are not optimized in terms of resource consumption. For example, some compression algorithms perform well in compression rate, but have great deficiencies in processing speed and computing resource consumption; while some encryption algorithms, although highly secure, have high computational complexity and are not suitable for real-time transmission and low-power devices. Therefore, it is necessary to research and develop new methods that can efficiently and reliably compress and encrypt and decrypt data under narrow bandwidth conditions.

[0005] At present, some progress has been made in the research of data compression encryption and decryption methods under low bandwidth conditions at home and abroad. The following is a summary of the current status of related research:

[0006] In the data compression and encryption method of the aviation system, the user encrypted data in the aviation system is too large, which leads to a decrease in the interaction speed. By generating a user string, constructing multiple character matrices and calculating the redundancy, the matrix with the largest redundancy is selected for Huffman coding to compress the data. Then, the encryption key length is determined according to the length of the compressed data, the compressed data is encrypted, and secondary encryption is achieved through random passwords and hash values. This method significantly improves the data compression rate and encryption strength, and enhances the efficiency and security of data transmission. The doping compression encryption method proposes a method of doping the encrypted data into a large amount of impurity data with certain repetitiveness and the same structure, and using a compression encryption method to dope the admixture with compression encryption. This method effectively reduces the data volume during the compression process by increasing the redundancy of the data, and enhances the complexity and security of encryption. The encryption and decryption module for secure transmission of large data proposes a preprocessing unit, an encryption unit at the sending end, a decryption unit and a post-processing unit at the receiving end. The module preprocesses and compresses the original data to be sent to reduce the amount of data to be encrypted. The encryption unit encrypts the data to ensure data security, the decryption unit decrypts the data at the receiving end, and the post-processing unit decompresses and restores the decrypted compressed data. This method improves transmission efficiency while ensuring data security. The segmented encryption method reads the file and encrypts it segment by segment. The parts that do not need to be kept confidential are not encrypted, and the parts that need to be encrypted use different encryption algorithms as needed. For existing keys or generated keys, the content that needs to be encrypted in the data block is first compressed and then encrypted. This method not only reduces the file size, but also ensures its security. It is suitable for situations where some file formats contain delimiters and length records.

[0007] The significant shortcomings of the above studies are as follows:

[0008] (1) Different data types have different statistical characteristics and redundancy patterns. Using a unified compression method ignores the characteristics of different data types and may lead to the following problems: low compression rate, unable to significantly reduce the amount of data; performance bottlenecks are prone to occur when processing large amounts of data or complex data types, increasing computing overhead and time costs; lack of flexibility, difficult to adapt to different application scenarios, especially in bandwidth-constrained environments; inability to fully compress data, resulting in a waste of bandwidth resources and reduced overall system efficiency.

[0009] (2) Data of different sensitivities require different encryption levels. Using a unified encryption level will reduce security and make it impossible to distinguish the sensitivity of data, increasing the risk of data leakage. At the same time, using high-intensity encryption for data of different sensitivities wastes computing resources, prolongs the encryption and decryption process, and affects system performance.

[0010] (3) Failure to effectively balance compression time, encryption time, transmission time, decryption time, and decompression time results in a total data processing and transmission time that is too long, increases the system's computing overhead and latency, and reduces overall efficiency and user experience. Summary of the invention

[0011] The purposes of the technical solution of the present invention are: 1. Data compression algorithm selection: Under low-bandwidth conditions, achieving efficient data compression is a key technical difficulty. Different types of data have different requirements for compression algorithms. It is very important to select different compression algorithms for different types of data. 2. Data encryption efficiency and security: Under low-bandwidth environments, encryption algorithms need to ensure both high efficiency (fast processing) and security, and can encrypt data at different levels based on the importance of the data. 3. Transmission delay and bandwidth utilization: Under low-bandwidth conditions, the delay in data transmission needs to be as low as possible, and the transmission time should be as short as possible to improve user experience.

[0012] The technical solution of the present invention provides a high-efficiency data compression encryption and decryption method under low bandwidth conditions, comprising the following steps:

[0013] The sender obtains multiple types of environmental data in real time, divides the environmental data into structured time series data and text data, and divides the structured time series data into integer data, floating point data, and Boolean data;

[0014] The Delta-of-Delta algorithm and Zig-Zag encoding are used for compression of integer data, the ZFP algorithm is used for compression of floating-point data, the Bit-Packing algorithm is used for compression of Boolean data, and the LZW algorithm is used for compression of text data;

[0015] Predefine a security policy based on the data source, content, and importance of the content in a specific application scenario, evaluate the importance of the compressed integer data and text data based on the predefined security policy, obtain the corresponding confidentiality level of the compressed integer data and text data, use an encryption algorithm combined with the confidentiality level to split and encrypt the compressed integer data and text data, and obtain an encrypted data stream as the output ciphertext;

[0016] Monitor the performance indicators of the data transmission network in real time, calculate the actual transmission speed of the current data transmission network or the transmission time required for a certain amount of data and conduct dynamic evaluation, and adjust the transmission strategy according to the dynamic evaluation results to transmit the output ciphertext to the receiver;

[0017] The receiver obtains the output ciphertext and matches the decryption algorithm corresponding to the output ciphertext to perform segmentation and decryption to obtain the decrypted data stream;

[0018] According to the compression algorithm corresponding to the decrypted data stream transmitted by the sender, if the compression algorithm is the Delta-of-Delta algorithm and the Zig-Zag encoding, the data difference sequence is restored by reverse calculation and the encoding conversion is restored to the integer data; if the compression algorithm is the ZFP algorithm, the floating-point data is gradually restored by reverse predictive encoding and quantization operations; if the compression algorithm is the Bit-Packing algorithm, the Boolean data is restored by unpacking the bitmap data; if the compression algorithm is the LZW algorithm, the text data is restored by restoring the compressed encoding sequence through reverse dictionary construction;

[0019] According to the original data sequence, data index information and structure information corresponding to the decrypted data stream transmitted by the sender, the decompressed decrypted data stream is restored to the original data structure, and the sequence of the structured time series data is restored according to the timestamp to obtain the reorganized data;

[0020] The decompressed decrypted data stream and the reassembled data are stored.

[0021] Preferably, when the sender performs the compression, corresponding compression algorithm parameters are set according to the selected algorithm.

[0022] Preferably, the encryption algorithm adopts AES algorithm.

[0023] Preferably, the split encryption process comprises the following steps:

[0024] Splitting the compressed integer data and text data into data blocks of preset sizes;

[0025] Convert the original key into a series of subkeys;

[0026] Perform an XOR operation on the data block with the subkey of the initial round to achieve the initial round encryption;

[0027] Byte substitution, row shift, column confusion and round key addition operations are performed in predetermined rounds until the final round of encryption performs byte substitution, row shift and round key addition operations to obtain an encrypted data stream as output ciphertext.

[0028] Preferably, the split decryption process comprises the following steps:

[0029] Split the output ciphertext into data blocks of preset size;

[0030] Perform an XOR operation on the data block with the subkey of the initial round to achieve initial round decryption;

[0031] The inverse byte substitution, reverse row shift, reverse column confusion and round key addition operations are performed in predetermined rounds until the final round of encryption performs the inverse byte substitution and reverse row shift operations to obtain a decrypted data stream.

[0032] Preferably, the efficient data compression encryption and decryption method comprises:

[0033] According to the total time T formula, calculate the optimal total time min(T):

[0034] T=T c (M)+T e (M′)+T t (C)+T d (C)+T u (M′)

[0035] Where M is the structured time series data, M′ is the compressed structured time series data that meets the distortion rate (minimum) requirement and M′ is also the decrypted data stream after decryption by the receiver, C is the encrypted data stream, T c is the compression time of M, T e is the encryption time of M′, T t is the transmission time of C, T d is the decryption time of C, T u is the decompression time of M′;

[0036] The sender starts to compress the structured time series data according to the optimal total time min(T), until the receiver decompresses the decrypted data stream to obtain the decompressed decrypted data stream;

[0037] The transmission strategy is adjusted according to the optimal total duration.

[0038] Preferably, the calculation of the optimal total duration min(T) according to the total duration T formula includes:

[0039] According to the compression and transmission relationship formula, the compression and encryption relationship formula, the compression and decompression relationship formula, and the decryption and compression relationship formula, combined with the total time length T formula, the final expression of the total time length T is obtained;

[0040] The minimize function is called on the final expression of the total duration T to perform nonlinear programming optimization to obtain the optimal total duration min(T).

[0041] Preferably, the compression and transmission relationship formula is:

[0042]

[0043] Among them, B represents the fluctuation of transmission time affected by bandwidth, k t represents compression and transmission constants;

[0044] The compression and decompression relationship formula:

[0045] T u =k u ×Tc

[0046] Among them, k u Represents compression and decompression constants.

[0047] Preferably, the compression and encryption relationship formula is:

[0048]

[0049] Among them, E is related to the encryption level, k e Represents compression and encryption constants;

[0050] The decryption and compression relationship formula:

[0051]

[0052] Among them, k d Represents decryption and compression constants.

[0053] Preferably, the optimization steps in calling the minimize function are as follows:

[0054] Calculate the gradient of the final expression of the total duration T relative to the compression time of the structured time series data, and update and iterate by moving in the direction of gradient descent to reduce the value of the final expression of the total duration T, and determine whether the change value of the compression time of the structured time series data reaches the preset threshold. If so, determine the minimum value of the compression time of the structured time series data.

[0055] The technical solution of the present invention proposes an efficient data compression encryption and decryption method under low bandwidth conditions. According to the characteristics of different data types, a specific compression algorithm is used to improve the compression rate and flexibility, and different encryption levels are used according to the sensitivity of the data to ensure security and resource utilization efficiency. At the same time, the technical solution of the present invention achieves a balance of overall time by optimizing the time of each link such as compression, encryption, transmission, decryption and decompression, thereby improving the overall efficiency of the system and user experience. The technical solution of the present invention effectively solves some problems currently existing in data compression and encryption and decryption under narrow bandwidth conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 An overall framework diagram provided for an embodiment of the present invention;

[0057] Figure 2 A data compression flow chart provided for an embodiment of the present invention;

[0058] Figure 3 AES encryption algorithm flow chart provided for an embodiment of the present invention;

[0059] Figure 4AES decryption algorithm flow chart provided by the embodiment of the present invention;

[0060] Figure 5 The data transmission volume and speed provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0062] An embodiment of the present invention provides a method for efficient data compression and encryption under low bandwidth conditions. The overall architecture of the method is as follows: Figure 1 As shown, the architecture is divided into two parts: sender and receiver.

[0063] The sender first collects various types of data generated, and then selects an appropriate compression algorithm to compress the data to reduce the size of the data. After the data compression is completed, the compressed data is encrypted to ensure the security of the data during transmission. Finally, the compressed and encrypted data is sent to the receiver through the network or other communication channels. The receiver receives the data transmitted by the sender through the network or other communication channels. The received data is encrypted ciphertext, so it is necessary to decrypt it to restore the original compressed data. After the data decryption is completed, the decrypted data is decompressed to restore the original uncompressed data. In the whole process, the time of each part is adaptively adjusted to achieve the optimal total duration. Finally, the receiver obtains the original data consistent with the data generated by the sender in the shortest time. The following is a detailed description of each step.

[0064] (1) Sender:

[0065] Step 1: Data generation.

[0066] Various sensors in different system devices collect environmental data in real time and transmit it to the central processing system, and users also transmit data text through terminal devices, so the data format ultimately sent by the sender is different.

[0067] Step 2: Data compression.

[0068] The data transmission process involves a variety of data types, including structured time series data, text data, etc. Different data types require different compression algorithms. For example, run length encoding is suitable for binary image compression, but not for compression of other data types. Through data compression, data can be represented in a more compact form, thereby saving storage space or transmission bandwidth and increasing the speed of data transmission. Therefore, it is necessary to select appropriate compression algorithms for different types of data, such as Figure 2 shown.

[0069] The steps of data compression are as follows:

[0070] ① Data classification: This includes classifying the collected data into structured time series data and text data. Structured time series data is further classified into integer data, floating point data, and Boolean data, while text data includes all unstructured text information, such as articles and logs.

[0071] ② Select compression algorithm: Select the most appropriate compression algorithm for different types of data. For integer data, applicable compression algorithms include Delta-of-Delta algorithm and Zig-Zag encoding, which can effectively reduce data redundancy. Floating-point data uses ZFP algorithm, which is an efficient algorithm specially designed for compressing floating-point numbers to improve compression efficiency. Boolean data compression often uses Bit-Packing technology, which can significantly reduce the space occupied by data. Text data usually uses LZW algorithm, which is a lossless compression algorithm that can significantly reduce the amount of data while ensuring data integrity. Compress the data according to the above compression algorithms and set the parameters of the compression algorithm, such as compression ratio and quality level, to ensure that the data can be transmitted efficiently when the bandwidth is limited.

[0072] ③Data aggregation: All types of compressed data are aggregated into a unified compressed data package. To ensure data integrity and simplify the decompression process, it is necessary to record the compression method of each data and its corresponding metadata, including data type, original size and other information.

[0073] Step 3: Data encryption.

[0074] The compressed data is classified according to its importance, and then the compressed data is encrypted according to the corresponding classification. The encryption algorithm selected in the embodiment of the present invention is the AES algorithm, and the detailed steps are as follows:

[0075] ① Confidentiality level setting: Based on the predefined security policy, the importance of the compressed data is evaluated. This step should be combined with factors such as the source, content and importance of the data in a specific application scenario. The data is divided into different confidentiality levels. The AES algorithm supports 128, 192 and 256-bit key lengths, corresponding to different confidentiality levels. The 128-bit key provides basic encryption security and is suitable for general data protection needs; the 192-bit key provides medium-strength encryption security and is suitable for scenarios that require higher security; and the 256-bit key provides the highest level of encryption security and is suitable for the protection of highly sensitive and confidential data.

[0076] ②Data encryption: Divide the compressed data into data blocks of appropriate size to adapt to the encryption processing of the AES algorithm. Taking 128-bit key encryption as an example, the algorithm encryption process is as follows: Figure 3 As shown. The encryption process mainly includes the following steps: First, key expansion, which converts the original key into a series of subkeys for use in different rounds of the encryption process. Next is the initial round encryption, at this time, the initial round key addition operation is performed on the data block, and the data block is processed by XOR operation with the initial round subkey. Then comes the main round encryption, which performs encryption operations according to the predetermined rounds, including byte substitution, row shift, column confusion and round key addition operations. The last round of encryption is called the final round encryption. In this round, the column confusion operation is no longer performed, but byte substitution, row shift and round key addition operations are directly performed. Finally, the encrypted ciphertext data block is integrated into a complete encrypted data stream as the output ciphertext.

[0077] (2) Data transmission:

[0078] During the data transmission process, the compressed and encrypted data needs to be packaged and encapsulated to meet the requirements of network transmission. In the application scenario of the embodiment of the present invention, the data transmission network presents the following characteristics: it has a wide coverage range and can provide communication services in areas that cannot be reached by the ground network. However, its bandwidth and stability are susceptible to the influence of the external environment. Performance will fluctuate due to factors such as weather conditions (such as rain, snow, thunderstorms) and solar magnetic storms, which may cause signal attenuation and increased interference, thereby affecting the reliability and speed of data transmission. Due to the above-mentioned environmental influences, the bandwidth and stability of the network will change frequently, and the transmission bandwidth may fluctuate greatly under different time periods and environmental conditions, affecting the efficiency of data transmission. In order to cope with these changes, it is necessary to monitor the performance indicators of the network in real time, including bandwidth, delay and packet loss rate. By detecting the network status, the transmission speed of the current network can be dynamically evaluated and the transmission strategy can be adjusted. According to the monitored network performance indicators, the actual transmission speed of the current network or the transmission time required for a certain amount of data can be converted. This evaluation helps to optimize the data transmission strategy and ensure that the optimal total duration of data transmission is achieved even when the network condition is poor.

[0079] (3) Recipient:

[0080] Step 1: Data decryption.

[0081] The receiver receives the data transmitted from the sender, and then decrypts the data according to the key. Taking 128-bit key encryption as an example, the algorithm decryption process is as follows Figure 4 As shown. The decryption operation mainly includes the following steps: perform an initial round key addition operation on each ciphertext data block, and perform an XOR operation on the ciphertext data block and the initial round key. The main round decryption includes inverse byte substitution, reverse row shift, reverse column confusion, and round key addition operations to gradually restore the original state of the data. In the final round of decryption, inverse byte substitution and reverse row shift operations are performed without column confusion, and the last round key addition operation is performed to restore the initial data block.

[0082] Step 2: Decompress the data.

[0083] Decompress the decrypted data. The detailed steps are as follows:

[0084] ① Compression algorithm identification: Determine the decompression algorithm corresponding to each type of data based on the compression method recorded in the metadata to ensure that the correct decompression algorithm is selected to process different types of data.

[0085] ② Data decompression: For integer data, the Delta-of-Delta and ZigZag decompression algorithms are used to restore the data difference sequence through reverse calculation and code conversion to restore the original data; for floating-point data, the ZFP decompression algorithm is used to gradually restore the original floating-point data through reverse predictive coding and quantization operations; for Boolean data, the Bit-Packing decompression algorithm is used to restore the original Boolean value sequence by unpacking the bitmap data; for text data, the LZW decompression algorithm is used to restore the compressed coding sequence through reverse dictionary construction to restore the complete text data.

[0086] ③Data restoration: Extract information about the original data sequence and data index from the metadata, including the location and sequence information of each data type. Then reorganize the decompressed data according to the sequence and index in the metadata, restore the reorganized data to the original data structure according to the structural information in the metadata, and restore the sequence of structured time series data according to timestamps or other identifiers.

[0087] Step 3: Data storage.

[0088] The process of saving the decompressed and reorganized data to a pre-specified database or data storage system. In this step, the reorganized data is stored persistently to ensure that subsequent applications and processing programs can correctly read and use the data. The goal of the data storage process is to ensure the long-term availability, consistency, and integrity of the data, thereby supporting the normal operation of the system and the realization of business needs.

[0089] (4) Optimal total duration:

[0090] In order to ensure the integrity and accuracy of the data, ensure the credibility and effectiveness of the information, and transmit the data to the target location as much as possible to support timely feedback and application response, the data must be sent to the other party in high quality and real time from the time it is generated. The time starts from the compression of the data and ends when the data is decompressed at the receiving end. The total time is:

[0091] T=T c (M)+T e (M ′ )+T t (C)+T d (C)+T u (M′) (1)

[0092] Among them, the data volume and speed of each step are as follows Figure 5 shown.

[0093] Figure 5 The area of ​​the box in represents the amount of data, while the long bar represents the processing time for compression / decompression, encryption / decryption, and data transmission. As can be seen from the formula and the figure, the various times are not independent of each other. If a compression algorithm or compression level with a high compression ratio is used, a higher compression ratio can be obtained, the amount of compressed data will be smaller, and the corresponding encryption and transmission time will also be shorter, but the compression time will be higher, because the compression algorithm will perform more rounds and deeper pattern discovery and aggregation, and the computational complexity is relatively high. Conversely, if the compression time is shorter, the compressed data will be larger, and the encryption and transmission time will be longer.

[0094] In addition, the time overhead coefficients for compressing, decompressing, encrypting, and decrypting the same amount of data are different, but compression and decompression are approximately equal, and encryption and decryption under the same key length are also approximately equal. Under different encryption levels, the higher the level, the longer the encryption and decryption time required. The transmission time overhead coefficient is the largest, especially in the satellite network environment, where the long signal transmission path leads to a significant increase in transmission delay. Moreover, the transmission time overhead coefficient is not fixed, but changes with changes in network conditions (such as bandwidth, delay, packet loss rate, etc.). The real-time transmission rate needs to be determined through network feedback. By monitoring network performance indicators in real time, the current transmission rate can be dynamically evaluated, and the transmission strategy can be adjusted accordingly to optimize the total data transmission time.

[0095] In short, the relationship between the various times is intertwined, and the influence of various factors needs to be comprehensively considered to achieve the overall optimum. The embodiment of the present invention uses nonlinear programming to find the optimal value of the total duration. The following are the constraints and relationships:

[0096] The relationship between compression and transmission: transmission time T t With compression time T C Inversely proportional, we can use a constant k t To describe this relationship, where B represents the fluctuation of transmission time affected by bandwidth:

[0097]

[0098] The relationship between compression and encryption: encryption time T e With compression time T C Inversely proportional, we can use a constant k e To describe, where E is related to the encryption level:

[0099]

[0100] The relationship between compression and decompression: compression time T C and decompression time T u is proportional to, and can be expressed as a constant k u To describe:

[0101] T u =k u ×T C (4)

[0102] Relationship between decryption and compression: decryption time T d With compression time T c Inversely proportional, we can use a constant k d To describe:

[0103]

[0104] Substituting formulas (2)-(5) into formula (1) yields the final expression for the total time T:

[0105]

[0106] According to formula (6), nonlinear programming optimization is performed and the minimize function is called to obtain the optimal total duration.

[0107] The minimize function uses a numerical optimization algorithm to iteratively search for the minimum value of the objective function under the initial guess and constraints. The optimization steps of the function are as follows:

[0108] Step 1: Calculate the gradient.

[0109] By calculating the objective function with respect to the variable T C The first-order derivative (gradient) of , the optimization algorithm can determine the search direction. The gradient represents the rate of change of the objective function at a specific point, and moving in the direction of gradient descent will reduce the value of the objective function. In each iteration, the algorithm calculates the gradient of the current point and uses this gradient information to update T C , thereby gradually reducing the value of the objective function.

[0110] Step 2: Iterative update.

[0111] Iterative updating is a repeated process where the algorithm computes the gradient and updates T at each step. C The value of makes the objective function gradually approach the minimum value. This process continues until the T that minimizes the objective function value is found. C .

[0112] Step 3: Convergence determination.

[0113] Convergence judgment means that during the optimization process, when the objective function value changes very little or reaches the preset convergence condition, the algorithm stops searching. When the change in the objective function value is lower than a certain threshold or reaches the maximum number of iterations, the algorithm considers that the optimal solution has been found and stops iterating. This mechanism ensures that the algorithm is completed within a reasonable time and avoids unlimited iterations.

[0114] An embodiment of the present invention also provides an efficient data compression and encryption and decryption device under low-bandwidth conditions, which adopts the above-mentioned efficient data compression and encryption and decryption method under low-bandwidth conditions, including: a total time optimization module, an efficient data compression module based on data classification, a data encryption module based on a predefined security policy, a data decryption module, and a data decompression and restoration module.

[0115] Among them, the total time optimization module uses the optimization method according to the current network performance to obtain the compression and encryption scheme with the shortest total time for compression, encryption, decryption, transmission and other links, and determines the compression method, compression level, encryption level and other parameters.

[0116] When planning a compression encryption and decryption solution, follow these steps:

[0117] a. Real-time monitoring of network performance indicators, including bandwidth, latency, packet loss rate and other information.

[0118] b. Dynamically evaluate the current network transmission speed and estimate the transmission time required to transmit a certain amount of data under the current network status.

[0119] c. Through nonlinear programming methods, the total delay of the entire process of data compression, encryption, transmission, decryption and decompression is used as the optimization indicator, the minimum security indicator, distortion rate, etc. are used as constraints, and the compression algorithm type, compression level, encryption level and other parameters are used as optimization objects. The optimization problem is solved to obtain the compression encryption and decryption scheme with the shortest total delay while satisfying the constraints.

[0120] The following steps are followed by nonlinear programming approach:

[0121] c1. Calculate the gradient: Calculate the first-order derivative (gradient) of the objective function with respect to the variable to determine the search direction.

[0122] c2, iterative update: repeat c1 and update the value of the total delay. During this process, the total delay gradually decreases until the compression encryption and decryption parameters that minimize the total delay are found.

[0123] c3. Convergence judgment: After each iteration, determine whether the number of cycles, optimization time, objective function value, etc. exceed the set threshold. If the threshold is reached, stop the iteration to ensure that the optimization algorithm is completed within a reasonable time and avoid unlimited iteration.

[0124] The efficient data compression module based on data classification classifies the data to be compressed according to the data type and adopts the optimal compression algorithm respectively, thereby improving the compression efficiency and generating compressed data packets.

[0125] To compress the original data, follow these steps:

[0126] (1) Data classification: Analyze the original data and first divide the data into structured time series data and text data. Then, further classify the structured time series data into integer data, floating point data, and Boolean data.

[0127] (2) Select compression algorithm: Select the optimal compression algorithm for different types of data. For example, use the Delta-of-Delta algorithm and Zig-Zag encoding for integer data, and the ZFP algorithm for floating-point data.

[0128] (3) Data aggregation: Aggregate all types of compressed data into a compressed package and record each data compression method and data type, original size, index and other information to form metadata.

[0129] The data encryption module based on the predefined security policy selects the encryption level and encryption method according to the type and degree of the data to be encrypted and the predefined encryption security policy, encrypts the compressed data packet, and generates an encrypted data packet.

[0130] Data encryption follows these steps:

[0131] A. Confidentiality level setting: Based on predefined security policies, the importance of data is evaluated and the encryption level of the data is determined.

[0132] B. Data encryption: Select the corresponding encryption algorithm for encryption according to the encryption level.

[0133] The data decryption module decrypts the encrypted data packet according to the corresponding encryption algorithm and restores the data packet to a compressed data packet.

[0134] The data decompression and restoration module decompresses the compressed data packet according to the corresponding compression algorithm, and then reorganizes and restores the decompressed data files separated by data type to obtain the original data.

[0135] To decompress and restore data, follow these steps:

[0136] s1. Compression algorithm identification: Determine the decompression algorithm corresponding to each type of data based on the compression method recorded in the metadata of the compressed data packet.

[0137] s2. Data decompression: decompress the compressed files in the compressed data packet one by one according to the corresponding decompression algorithm to obtain several data files classified by data type.

[0138] s3. Extract index information from metadata, restore according to corresponding structure and timing information, and restore several decompressed data files to original files.

[0139] The key technical points of the embodiments of the present invention include selecting the most suitable compression algorithm according to the data type, using different encryption levels according to the importance of the data, and optimizing the total transmission time through nonlinear programming methods. These innovations effectively improve the efficiency and security of data transmission, and are particularly suitable for data transmission requirements under low bandwidth conditions.

[0140] The embodiment of the present invention adopts a variety of data compression algorithms to improve data transmission efficiency and security. These algorithms together constitute the core technology of the invention. The algorithms adopted by the embodiment of the present invention will be described in detail below:

[0141] (1) Delta of Delta:

[0142] Delta of Delta, also known as second-order differential coding, is a method of using Delta coding again on the basis of Delta coding. It is more suitable for coding monotonically increasing or decreasing sequence data. For example, 2,4,4,6,8, after Delta coding, becomes 2,2,0,2,2, and after Delta coding again, becomes 2,0,-2,0,0.

[0143] Facebook Gorilla uses delta-of-delta coding to calculate and provides a more general processing solution for data of different time spans (a+b=c, a represents the storage flag bit occupies a bit, b represents the bit required to store data), as shown in Table 1. The compression capability of delta-of-delta coding is shown in Table 2.

[0144] Table 1 General treatment plan

[0145] Data Range Identification bit Total bits occupied 0 0 1 [-63，64] 10 2+7＝9 [-255，256] 110 3+9＝12 [-2047，2048] 1110 4+12＝16 >2048 1111 4+32＝36

[0146] Table 2 Compression capability of delta-of-delta encoding

[0147]

[0148]

[0149] Assuming the start timestamp is 1571889600000 and the maximum range threshold of delta is 3600s, the comparison of occupied storage space is as follows:

[0150] Delta-of-delta algorithm: 64+9*4+1*3=103bit. It can be seen that the delta-of-delta algorithm has a higher compression rate than the delta algorithm. In actual application scenarios, the timestamps of massive time series data are dense and continuous, and most of them meet the condition of delta-of-delta=0, which can greatly reduce the storage space of timestamps.

[0151] Compared with the uncompressed original timestamp data generated within 3600s, if one piece of data is generated every 3ms, the total number of bits is: 3600*1000 / 3*64=745kbit, and the compression rate is about 1 / 7000.

[0152] (2) Zig-Zag algorithm:

[0153] The first sign bit of the complement code is placed at the end of the complement code, and the other bits are moved forward one position as a whole; all the data bits of a negative number are inverted bit by bit, and the sign bit remains unchanged; for non-negative integers, the sign bit is moved to the end, and the other bits are moved forward one position, and the data remains unchanged.

[0154] (3) ZFP algorithm:

[0155] The steps of the ZFP compression algorithm are as follows:

[0156] Step 1: The d-dimensional array is split into several 4-dimensional arrays. d The d-dimensional array is called a block. If the d-dimensional array cannot be exactly divided into blocks, the boundaries of the d-dimensional array are padded until it can be exactly divided.

[0157] Step 2: Convert the floating-point values ​​in each block to block floating-point representation: Convert all numbers in the block to signed binary format, and extract the exponent of the data with the largest absolute value in the block as the public exponent. In this way, all floating-point numbers in the block are represented by binary integers and a public exponent.

[0158] Step 3: Perform a high-performance customized decorrelation operation on the integers through a method similar to the discrete cosine transform. This method is similar to the discrete cosine transform, which is to reduce the correlation of the data through a series of mathematical operations on the transformation matrix. The reason for the decorrelation operation is that continuous fields often show autocorrelation, which can be regarded as redundant information. Through the decorrelation operation, the data can be made more dispersed after the transformation, so that fewer bits are used to represent the same or similar information, which improves the efficiency of compression.

[0159] Step 4: Sort the coefficients in order of decreasing magnitude: The magnitude of the coefficients is often related to the sequence (inversely proportional). The purpose of sequence sorting is to arrange the coefficients roughly in order of decreasing magnitude, so that 1s and 0s are grouped together on each bit plane. This is because the coefficients at the front are usually larger non-zero coefficients, while the smaller coefficients at the back are often zero or close to zero. Through this step, most of the smaller coefficients will be placed at the end of the sequence, and small coefficients usually share the same number of leading zeros, which enhances zero clustering. Leading zeros can be encoded in a more compact form, thereby reducing the amount of compressed data.

[0160] Step 5: Convert the two's complement signed integer to negative binary: The sign bit is usually the leftmost bit in any traditional binary representation, and it does not provide any useful information until a leading bit is encountered (i.e., a transition from 0 to 1 (positive value) or from 1 to 0 (negative value)). However, the first "1" in the negative binary representation reflects the sign and value range of the entire number. For example, if the leftmost bit of the negative binary is in the e position, then the magnitude of the number is between 2 e · In the range [1 / 3, 4 / 3]. If e is even, then the number is positive; otherwise, it is negative. Therefore, the two's complement signed integer (standard integer representation) is converted to negative binary representation. Negative binary representation also concentrates errors introduced by calculation or processing near zero rather than spreading them out over the entire range of values.

[0161] Step 6: Pair 4 d The bits of the list of integers are transposed so that they are sorted by bit plane (most significant bit to least significant bit) rather than by coefficient.

[0162] Step 7: Lossless compression of coefficients using embedded coding: This coding method divides each bit plane into two parts in view of the fact that there are many leading zeros in the transform coefficients: First, the n lowest bits are directly output without coding, where n is dynamic and is determined by the number of bits that have been coded to ensure that the high 4 bits of all previous bit planes are d -n bits are all zero, and n is set to 0 during initialization; then, the remaining bits are encoded using unary run length coding. Finally, the compressed code of each bit plane is output.

[0163] Step 8: Output compressed data: Through the previous operations of dividing, converting, sorting and embedded coding of the input data, an encoded bit stream is finally generated. The previously generated data is sent bit by bit through the embedded encoder until the stopping criterion is met. The embedded encoder is still used in this step because the embedded encoder adopts an adaptive encoding method, which can adjust the encoding method more flexibly according to the previous encoded bits to minimize the length of the compressed bit stream. Bit-by-bit transmission can improve accuracy and compression efficiency while maintaining truncation flexibility and adapting to the distribution of transform coefficients. The specific stopping criterion depends on the ZFP compression mode: fixed rate, fixed precision or fixed accuracy.

[0164] (4) Bit-Packing algorithm:

[0165] Boolean type usually occupies 8 bits in most programming languages, and represents false when it is 0, and represents true when it is 1 (or greater than 1). Bit-Packing represents Boolean array as byte array, and the operation steps are as follows:

[0166] ① Byte index: floor(i / 8);

[0167] ②bit index: i mod 8;

[0168] ③Bit shift operation, merge after shifting. For example: a Boolean array contains 4 Boolean data: 00000001 00000010 00000011 00000000

[0169] After shifting and merging, we get: 00111001

[0170] When decoding, you only need to perform the reverse operation.

[0171] The above example uses one byte to represent the encoding result. For longer Boolean arrays, multiple bytes or byte arrays are required to represent them.

[0172] Compression rate analysis: Boolean data occupies one byte (8 bits), but only the last bit of 0 or 1 is actually valid. Therefore, when compressing Boolean data, the compression rate is very high when using Bit-Packing: 8 Boolean values ​​originally occupy 8 bytes (64 bits), but after Bit-Packing compression, only 1 byte (8 bits) is required, with a compression rate of up to 12.5%.

[0173] (5) LZW algorithm:

[0174] The LZW compression algorithm achieves compression by building a dictionary. First, a single character is added to the initial dictionary, and then the input data is scanned and the dictionary is gradually built, replacing the existing strings with indexes. During the scanning process, when a string that does not exist in the dictionary is encountered, it is added to the dictionary and a new index is assigned. This process is repeated until the scan is completed. By outputting the index sequence of the strings in the dictionary, the LZW algorithm can compress the input data losslessly and achieve good compression effects.

[0175] The embodiment of the present invention is divided into two parts: a sender and a receiver. The sender includes data generation, compression, encryption and transmission, wherein the data compression part can select different compression algorithms according to different data types, and the data encryption part can select different encryption levels according to the importance of the data; the receiver includes receiving data, decryption, decompression and storage. In order to obtain the optimal total transmission time, the nonlinear programming method is used to optimize the compression time, encryption time, transmission time, decryption time and decompression time.

[0176] In summary, the main beneficial effects and contributions of the embodiments of the present invention can be summarized as follows:

[0177] (1) When compressing data, the most appropriate compression algorithm is selected according to the data type, thereby improving compression efficiency and compression rate, significantly reducing the amount of data, saving bandwidth, and optimizing transmission performance.

[0178] (2) When encrypting data, different encryption levels are selected according to the importance of the data, thereby reducing the time and computing resource consumption for encryption and decryption while ensuring the necessary security, thereby improving the security and efficiency of the system.

[0179] (3) Through nonlinear programming methods, the time relationship between each stage of data compression, encryption, transmission, decryption and decompression is optimized, and the optimal total duration is obtained under variable conditions such as compression algorithm selection, compression level, encryption level, and transmission conditions. This ensures that data can be transmitted efficiently, quickly, and securely under narrow bandwidth conditions, thereby improving the overall efficiency and performance of data transmission.

Claims

1. A high-efficiency data compression encryption and decryption method under low bandwidth conditions, characterized in that: The following steps are involved: The sender obtains multiple types of environmental data in real time, divides the environmental data into structured time series data and text data, and divides the structured time series data into integer data, floating point data, and Boolean data; The Delta-of-Delta algorithm and Zig-Zag encoding are used for compression of integer data, the ZFP algorithm is used for compression of floating-point data, the Bit-Packing algorithm is used for compression of Boolean data, and the LZW algorithm is used for compression of text data; Predefine a security policy based on the data source, content, and importance of the content in a specific application scenario, evaluate the importance of the compressed integer data and text data based on the predefined security policy, obtain the corresponding confidentiality level of the compressed integer data and text data, use an encryption algorithm combined with the confidentiality level to split and encrypt the compressed integer data and text data, and obtain an encrypted data stream as the output ciphertext; Monitor the performance indicators of the data transmission network in real time, calculate the actual transmission speed of the current data transmission network or the transmission time required for a certain amount of data and conduct dynamic evaluation, and adjust the transmission strategy according to the dynamic evaluation results to transmit the output ciphertext to the receiver; The receiver obtains the output ciphertext and matches the decryption algorithm corresponding to the output ciphertext to perform segmentation and decryption to obtain the decrypted data stream; According to the compression algorithm corresponding to the decrypted data stream transmitted by the sender, if the compression algorithm is the Delta-of-Delta algorithm and the Zig-Zag encoding, the data difference sequence is restored by reverse calculation and the encoding conversion is restored to the integer data; if the compression algorithm is the ZFP algorithm, the floating-point data is gradually restored by reverse predictive encoding and quantization operations; if the compression algorithm is the Bit-Packing algorithm, the Boolean data is restored by unpacking the bitmap data; if the compression algorithm is the LZW algorithm, the text data is restored by restoring the compressed encoding sequence through reverse dictionary construction; According to the original data sequence, data index information and structure information corresponding to the decrypted data stream transmitted by the sender, the decompressed decrypted data stream is restored to the original data structure, and the sequence of the structured time series data is restored according to the timestamp to obtain the reorganized data; The decompressed decrypted data stream and the reassembled data are stored.

2. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 1, characterized in that: When the sender performs the compression, corresponding compression algorithm parameters are set according to the selected algorithm.

3. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 1, characterized in that: The encryption algorithm adopts AES algorithm.

4. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 1, characterized in that: The split encryption process includes the following steps: Splitting the compressed integer data and text data into data blocks of preset sizes; Convert the original key into a series of subkeys; Perform an XOR operation on the data block and the subkey of the initial round to achieve the initial round encryption; Byte substitution, row shift, column confusion and round key addition operations are performed in predetermined rounds until the final round of encryption performs byte substitution, row shift and round key addition operations to obtain an encrypted data stream as output ciphertext.

5. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 1, characterized in that: The process of segmentation and decryption comprises the following steps: Split the output ciphertext into data blocks of preset size; Perform an XOR operation on the data block with the subkey of the initial round to achieve initial round decryption; The inverse byte substitution, reverse row shift, reverse column confusion and round key addition operations are performed in predetermined rounds until the final round of encryption performs the inverse byte substitution and reverse row shift operations to obtain a decrypted data stream.

6. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 1, characterized in that: The efficient data compression encryption and decryption method comprises: According to the total time T formula, calculate the optimal total time min(T): T=T c (M)+T e (M ′ )+T t (C)+T d (C)+T u (M′) Where M is the structured time series data, M′ is the compressed structured time series data that meets the distortion rate (minimum) requirement and M′ is also the decrypted data stream after decryption by the receiver, C is the encrypted data stream, T c is the compression time of M, T e is the encryption time of M', T t is the transmission time of C, T d is the decryption time of C, T u is the decompression time of M′; The sender starts to compress the structured time series data according to the optimal total time min(T), until the receiver decompresses the decrypted data stream to obtain the decompressed decrypted data stream; The transmission strategy is adjusted according to the optimal total duration.

7. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 6, characterized in that: The method of calculating the optimal total duration min(T) according to the total duration T formula includes: According to the compression and transmission relationship formula, the compression and encryption relationship formula, the compression and decompression relationship formula, and the decryption and compression relationship formula, combined with the total time length T formula, the final expression of the total time length T is obtained; The minimize function is called on the final expression of the total duration T to perform nonlinear programming optimization to obtain the optimal total duration min(T).

8. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 7, characterized in that: The compression and transmission relationship formula: Among them, B represents the fluctuation of transmission time affected by bandwidth, k t represents compression and transmission constants; The compression and decompression relationship formula: T u =k u ×T C Among them, k u Represents compression and decompression constants.

9. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 7, characterized in that: The relationship formula between compression and encryption: Among them, E is related to the encryption level, k e Represents compression and encryption constants; The decryption and compression relationship formula: Among them, k d Represents decryption and compression constants.

10. The high-efficiency data compression encryption and decryption method under low bandwidth conditions as claimed in claim 7, characterized in that: The optimization steps in calling the minimize function are as follows: Calculate the gradient of the final expression of the total duration T relative to the compression time of the structured time series data, and update and iterate by moving in the direction of gradient descent to reduce the value of the final expression of the total duration T, and determine whether the change value of the compression time of the structured time series data reaches the preset threshold. If so, determine the minimum value of the compression time of the structured time series data.

Images