Efficient data compression and decryption method under low bandwidth condition

Abstract

The application provides a high-efficiency data compression and decryption method under a low bandwidth condition, adopts specific compression algorithms according to the characteristics of different data types to improve the compression rate and flexibility, and adopts different encryption levels according to the data sensitivity to ensure the security and resource utilization efficiency.Meanwhile, the application balances the overall time by optimizing the time of compression, encryption, transmission, decryption and decompression and other links, thereby improving the overall efficiency and user experience of the system.The application effectively solves some problems existing in the data compression and decryption under the narrow bandwidth condition.

Description

Technical Field

[0001] This invention relates to the fields of data compression and encryption technology, communication technology and information security technology, and specifically to an efficient data compression and encryption / decryption method under low bandwidth conditions. Background Technology

[0002] In modern communication networks, many practical applications suffer from bandwidth limitations, such as communication between ships and shore, network connectivity in remote areas, IoT devices, wireless sensor networks, and satellite communications. The devices and network infrastructure in these scenarios typically cannot provide high bandwidth support, thus requiring efficient data compression and encryption / decryption methods to improve transmission efficiency and security.

[0003] Under limited bandwidth conditions, using a single compression algorithm may not achieve the desired results. Therefore, different compression algorithms need to be selected for different types of data to improve compression ratio and ensure data quality. Meanwhile, with the increase in cyberattacks and growing awareness of data privacy, the importance of data encryption technology is becoming increasingly prominent. Advanced encryption algorithms can ensure efficient and secure data transmission even under limited bandwidth conditions.

[0004] Many current compression and encryption technologies perform well in high-bandwidth, high-performance computing environments, but their performance is subpar in narrow-bandwidth, low-power applications. Existing technologies often cannot simultaneously meet the requirements of efficient compression and secure encryption, or are not optimized in terms of resource consumption. For example, some compression algorithms perform well in terms of compression ratio, but have significant shortcomings in processing speed and computational resource consumption; while some encryption algorithms, although highly secure, have high computational complexity and are not suitable for real-time transmission and low-power devices. Therefore, it is necessary to research and develop new methods that can perform data compression and encryption / decryption efficiently and reliably under narrow-bandwidth conditions.

[0005] Currently, significant progress has been made in the research of data compression and decryption methods under low bandwidth conditions, both domestically and internationally. The following is a summary of the current state of related research:

[0006] In aviation systems, data compression and encryption methods address the issue of reduced interaction speed caused by excessively large user encrypted data. One approach involves generating user strings, constructing multiple character matrices, calculating redundancy, and selecting the matrix with the highest redundancy for Huffman coding to compress the data. Then, the encryption key length is determined based on the compressed data length, and the compressed data is encrypted. A second encryption is achieved using a random password and hash value. This method significantly improves data compression rate and encryption strength, enhancing data transmission efficiency and security. A doping compression and encryption method proposes incorporating the data to be encrypted into a large amount of repetitive and structurally identical impurity data, using compression and encryption methods to dope and compress the mixture. This method effectively reduces data volume during compression by increasing data redundancy, enhancing encryption complexity and security. A big data secure transmission encryption and decryption module includes a preprocessing unit and an encryption unit at the sending end, and a decryption unit and a post-processing unit at the receiving end. This module preprocesses and compresses the original data to be sent, reducing the amount of data to be encrypted. The encryption unit encrypts the data to ensure data security, the decryption unit decrypts the data at the receiving end, and the post-processing unit decompresses and restores the decrypted compressed data. This method improves transmission efficiency while ensuring data security. The segmented encryption method reads the file and encrypts it in segments; parts that do not need to be kept confidential are left unencrypted, while the parts that need encryption are encrypted using different encryption algorithms as needed. Using an existing or generated key, the content to be encrypted in the data block is first compressed before encryption. This method not only reduces file size but also ensures its security, and is suitable for file formats containing delimiters and length records.

[0007] The significant drawbacks of the above research are as follows:

[0008] (1) Different data types have different statistical characteristics and redundancy patterns. Using a uniform compression method ignores the characteristics of different data types, which may lead to the following problems: low compression rate, unable to significantly reduce the amount of data; performance bottlenecks are likely to occur when processing large amounts of data or complex data types, increasing computational overhead and time costs; lack of flexibility, making it difficult to adapt to different application scenarios, especially in bandwidth-limited environments; and failure to fully compress data, resulting in wasted bandwidth resources and reduced overall system efficiency.

[0009] (2) Different levels of data sensitivity require different levels of encryption. Using a uniform encryption level will reduce security, make it impossible to distinguish the sensitivity of data, and increase the risk of data leakage. At the same time, using high-strength encryption for different levels of data sensitivity wastes computing resources, prolongs the encryption and decryption process, and affects system performance.

[0010] (3) Failure to effectively balance compression time, encryption time, transmission time, decryption time and decompression time resulted in excessively long total data processing and transmission time, increasing the system's computational overhead and latency, and reducing overall efficiency and user experience. Summary of the Invention

[0011] The objectives of this invention are as follows: 1. Data compression algorithm selection: Achieving efficient data compression under low bandwidth conditions is a key technical challenge. Different types of data have different requirements for compression algorithms, making it crucial to select different compression algorithms for different types of data. 2. Data encryption efficiency and security: In low bandwidth environments, encryption algorithms need to ensure both high efficiency (fast processing) and security, and should be able to apply different levels of encryption based on the importance of the data. 3. Transmission latency and bandwidth utilization: Under low bandwidth conditions, data transmission latency needs to be as low as possible, and transmission time should be as short as possible to improve user experience.

[0012] The present invention provides an efficient data compression and encryption method under low bandwidth conditions, comprising the following steps:

[0013] The sender acquires various types of environmental data in real time, classifying the environmental data into structured time-series data and text data, and further classifying the structured time-series data into integer data, floating-point data, and Boolean data.

[0014] For integer data, the Delta-of-Delta algorithm and Zig-Zag encoding are used for compression; for floating-point data, the ZFP algorithm is used for compression; for Boolean data, the Bit-Packing algorithm is used for compression; and for text data, the LZW algorithm is used for compression.

[0015] Based on the data source, content, and importance of the content in a specific application scenario, a security policy is predefined. The importance of the compressed integer data and text data is evaluated according to the predefined security policy to obtain the corresponding security level of the compressed integer data and text data. An encryption algorithm is used in combination with the security level to segment and encrypt the compressed integer data and text data, and the encrypted data stream is used as the output ciphertext.

[0016] Real-time monitoring of data transmission network performance indicators, calculation of the actual transmission speed of the current data transmission network or the transmission time required for a certain amount of data and dynamic evaluation, and adjustment of transmission strategy based on dynamic evaluation results to transmit the output ciphertext to the receiver.

[0017] The receiver obtains the output ciphertext and matches it with the corresponding decryption algorithm to segment and decrypt it, thus obtaining the decrypted data stream.

[0018] Based on the compression algorithm corresponding to the decrypted data stream transmitted by the sender, if the compression algorithm is Delta-of-Delta and Zig-Zag encoding, then the integer data is restored by reverse calculation of the data difference sequence and encoding conversion; if the compression algorithm is ZFP, then the floating-point data is gradually restored by reverse prediction encoding and quantization operations; if the compression algorithm is Bit-Packing, then the Boolean data is restored by unpacking the bitmap data; if the compression algorithm is LZW, then the text data is restored by constructing a reverse dictionary to restore the compressed encoding sequence.

[0019] Based on the original data order, data index information, and structure information corresponding to the decrypted data stream transmitted by the sender, the decompressed data stream is restored to its original data structure, and the order of the structured time-series data is restored according to the timestamp to obtain the recombined data.

[0020] The decompressed data stream and the reassembled data are stored.

[0021] Preferably, when the sender performs the compression, the corresponding compression algorithm parameters are set according to the selected algorithm.

[0022] Preferably, the encryption algorithm is the AES algorithm.

[0023] Preferably, the segmentation and encryption process includes the following steps:

[0024] The compressed integer and text data are divided into data blocks of a preset size;

[0025] Convert the original key into a series of subkeys;

[0026] Perform an XOR operation between the data block and the subkey of the initial round to achieve initial round encryption;

[0027] The byte substitution, row shift, column obfuscation, and round key addition operations are performed in predetermined rounds until the final round of encryption, which performs byte substitution, row shift, and round key addition operations to obtain an encrypted data stream as the output ciphertext.

[0028] Preferably, the segmentation and decryption process includes the following steps:

[0029] The output ciphertext is divided into data blocks of a preset size;

[0030] Perform an XOR operation between the data block and the subkey of the initial round to achieve initial round decryption;

[0031] The process involves performing reverse byte substitution, reverse row shift, reverse column obfuscation, and round key addition operations in predetermined rounds until the final round of encryption performs reverse byte substitution and reverse row shift operations to obtain the decrypted data stream.

[0032] Preferably, the efficient data compression and encryption / decryption method includes:

[0033] The optimal total duration min(T) is calculated using the formula for total duration T:

[0034] T = T c (M)+T e (M′)+T t (C)+T d (C)+T u (M′)

[0035] Where M is structured time-series data, M′ is compressed structured time-series data that meets the minimum distortion rate requirement, and M′ is also the decrypted data stream after decryption by the receiver, C is encrypted data stream, and T... c The compression time of M, T e T is the encryption time of M′. t T is the transmission time of C. d It is the decryption time of C, T u This is the decompression time of M′;

[0036] The timer for the sender to compress the structured time-series data is calculated based on the optimal total duration min(T) until the receiver decompresses the decrypted data stream to obtain the decompressed decrypted data stream.

[0037] The transmission strategy is adjusted based on the optimal total duration.

[0038] Preferably, the step of calculating the optimal total duration min(T) according to the formula for total duration T includes:

[0039] Based on the formulas for the relationship between compression and transmission, compression and encryption, compression and decompression, and decryption and compression, combined with the formula for total duration T, the final expression for total duration T is obtained.

[0040] The optimal total duration min(T) is obtained by performing nonlinear programming optimization on the final expression of the total duration T by calling the minimize function.

[0041] Preferably, the formula for the relationship between compression and transmission is:

[0042]

[0043] Where B represents the fluctuation in transmission time due to bandwidth, and k t Represents compression and transmission constants;

[0044] The formula relating compression and decompression is as follows:

[0045] T u =k u ×Tc

[0046] Where, k u This represents the compression and decompression constants.

[0047] Preferably, the formula for the relationship between compression and encryption is:

[0048]

[0049] Where E represents the encryption level, k e Represents compression and encryption constants;

[0050] The formula relating decryption and compression is as follows:

[0051]

[0052] Where, k d This represents the decryption and compression constants.

[0053] Preferably, the optimization steps in the call to the minimize function are as follows:

[0054] The gradient of the final expression of the total duration T relative to the compression time of the structured time series data is calculated. The value of the final expression of the total duration T is updated and iterated by moving along the direction of gradient descent to reduce the value of the final expression of the total duration T. It is then determined whether the change in the compression time of the structured time series data reaches a preset threshold. If it does, the minimum value of the compression time of the structured time series data is determined.

[0055] This invention proposes a high-efficiency data compression and encryption / decryption method under low bandwidth conditions. Specific compression algorithms are employed to improve compression ratio and flexibility, tailored to the characteristics of different data types. Different encryption levels are used based on data sensitivity to ensure security and resource utilization efficiency. Furthermore, this invention optimizes the time required for each stage of compression, encryption, transmission, decryption, and decompression, achieving an overall time balance and improving overall system efficiency and user experience. This invention effectively solves some of the problems existing in current data compression and encryption / decryption under narrow bandwidth conditions. Attached Figure Description

[0056] Figure 1 This is an overall framework diagram provided for an embodiment of the present invention;

[0057] Figure 2 A data compression flowchart provided for embodiments of the present invention;

[0058] Figure 3 A flowchart of the AES encryption algorithm provided in an embodiment of the present invention;

[0059] Figure 4A flowchart of the AES decryption algorithm provided in this embodiment of the invention;

[0060] Figure 5 The data transmission volume and speed provided in the embodiments of the present invention. Detailed Implementation

[0061] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0062] This invention provides an efficient data compression and encryption / decryption method under low bandwidth conditions. The overall architecture of the method is as follows: Figure 1 As shown, the architecture is divided into two parts: the sender and the receiver.

[0063] The sender first collects various types of data and then selects an appropriate compression algorithm to compress the data to reduce its size. After compression, the compressed data is encrypted to ensure security during transmission. Finally, the compressed and encrypted data is sent to the receiver via a network or other communication channel. The receiver receives the data transmitted by the sender via the network or other communication channel. The received data is encrypted ciphertext, so it needs to be decrypted to recover the original compressed data. After decryption, the decrypted data is decompressed to recover the original uncompressed data. Throughout the process, the time of each part is adaptively adjusted to achieve the optimal total duration. Ultimately, the receiver obtains the original data identical to that generated by the sender in the shortest possible time. The following is a detailed explanation of each step.

[0064] (1) Sender:

[0065] Step 1: Data generation.

[0066] Various sensors in different system devices collect environmental data in real time and transmit it to the central processing system. Users also transmit data text through terminal devices. Therefore, the data format sent by the sender is different.

[0067] Step 2: Data compression.

[0068] Data transmission involves various data types, including structured time-series data and text data. Different data types require different compression algorithms. For example, run-length encoding is suitable for binary image compression but not for other data types. Data compression allows data to be represented in a more compact form, thereby saving storage space or transmission bandwidth and improving data transmission speed. Therefore, it is necessary to select appropriate compression algorithms for different types of data. Figure 2 As shown.

[0069] The steps for data compression are as follows:

[0070] ① Data Classification: This includes classifying the collected data into structured time-series data and text data. Structured time-series data is further classified into integer data, floating-point data, and Boolean data, while text data includes all unstructured text information, such as articles and logs.

[0071] ② Select a compression algorithm: Choose the most suitable compression algorithm for different types of data. For integer data, suitable compression algorithms include Delta-of-Delta and Zig-Zag encoding, which can effectively reduce data redundancy. Floating-point data uses the ZFP algorithm, a highly efficient algorithm specifically designed for compressing floating-point numbers to improve compression efficiency. Boolean data compression often uses bit-packing technology, which can significantly reduce the data's footprint. Text data typically uses the LZW algorithm, a lossless compression algorithm that can significantly reduce data size while maintaining data integrity. Compress the data according to the above compression algorithms and set the compression algorithm parameters, such as compression ratio and quality level, to ensure efficient data transmission under bandwidth constraints.

[0072] ③ Data aggregation: All compressed data is aggregated into a single compressed data package. To ensure data integrity and simplify the decompression process, it is necessary to record the compression method and corresponding metadata for each type of data, including data type, original size, and other information.

[0073] Step 3: Data encryption.

[0074] The compressed data is assigned a security level based on its importance, and then encrypted according to the corresponding security level. The encryption algorithm selected in this embodiment of the invention is the AES algorithm, and the detailed steps are as follows:

[0075] ① Security Classification: Based on predefined security policies, the importance of the compressed data is assessed. This step considers factors such as the data's source, content, and its importance in specific application scenarios. The data is divided into different security classes. The AES algorithm supports 128, 192, and 256-bit key lengths, each corresponding to a different security class. A 128-bit key provides basic encryption security, suitable for general data protection needs; a 192-bit key provides medium-strength encryption security, suitable for scenarios requiring higher security; and a 256-bit key provides the highest level of encryption security, suitable for protecting highly sensitive and confidential data.

[0076] ② Data Encryption: The compressed data is divided into appropriately sized blocks to accommodate the AES encryption algorithm. Taking 128-bit key encryption as an example, the encryption process is as follows: Figure 3 As shown, the encryption process mainly includes the following steps: First, key expansion, which transforms the original key into a series of subkeys for use in different rounds of the encryption process. Next is the initial round encryption, where the data block is processed by XORing the initial round key with the initial round subkeys. Then comes the main round encryption, which performs encryption operations according to a predetermined sequence, including byte substitution, row shifting, column obfuscation, and round key addition. The final round encryption is called the final round encryption. In this round, column obfuscation is not performed; instead, byte substitution, row shifting, and round key addition are performed directly. Finally, the encrypted ciphertext data blocks are integrated into a complete encrypted data stream, which serves as the output ciphertext.

[0077] (2) Data transmission:

[0078] During data transmission, compressed and encrypted data needs to be packaged and encapsulated to meet network transmission requirements. In the application scenario of this invention, the data transmission network exhibits the following characteristics: it has a wide coverage area, enabling communication services in areas inaccessible to terrestrial networks. However, its bandwidth and stability are susceptible to external environmental influences. Performance fluctuates due to weather conditions (such as rain, snow, and thunderstorms) and solar magnetic storms, which can lead to signal attenuation and increased interference, thus affecting the reliability and speed of data transmission. Due to these environmental influences, network bandwidth and stability change frequently, and transmission bandwidth may fluctuate significantly under different time periods and environmental conditions, affecting data transmission efficiency. To address these changes, it is necessary to monitor network performance indicators in real time, including bandwidth, latency, and packet loss rate. By monitoring network conditions, the current network transmission speed can be dynamically assessed, and transmission strategies can be adjusted. Based on the monitored network performance indicators, the actual transmission speed of the current network or the transmission time required for a certain amount of data can be calculated. This assessment helps optimize data transmission strategies, ensuring that the optimal total data transmission time is achieved even under poor network conditions.

[0079] (3) Recipient:

[0080] Step 1: Data decryption.

[0081] The receiver receives the data transmitted from the sender and then decrypts the data using the key. Taking 128-bit key encryption as an example, the decryption process is as follows: Figure 4 As shown. The decryption operation mainly includes the following steps: Performing an initial round key addition operation on each ciphertext data block, and then XORing the ciphertext data block with the initial round key. The main round decryption includes inverse byte substitution, reverse row shifting, inverse column obfuscation, and round key addition operations, gradually restoring the original state of the data. In the final round decryption, inverse byte substitution and reverse row shifting operations are performed, column obfuscation is not performed, and a final round key addition operation is performed to restore the initial data block.

[0082] Step 2: Data decompression.

[0083] The decompressed data is then decompressed. The detailed steps are as follows:

[0084] ① Compression algorithm identification: Based on the compression method recorded in the metadata, determine the decompression algorithm corresponding to each type of data to ensure that the correct decompression algorithm is selected to process different types of data.

[0085] ② Data decompression: For integer data, Delta-of-Delta and ZigZag decompression algorithms are used to recover the data difference sequence and restore the original data through reverse calculation and encoding conversion; for floating-point data, ZFP decompression algorithm is used to gradually recover the original floating-point data through reverse predictive coding and quantization operations; for Boolean data, Bit-Packing decompression algorithm is used to recover the original Boolean value sequence by unpacking bitmap data; for text data, LZW decompression algorithm is used to restore the compressed encoding sequence by constructing a reverse dictionary to recover the complete text data.

[0086] ③ Data Restoration: Extract information about the original data order and data index from the metadata, including the position and order information of each data type. Then, reassemble the various types of data after decompression according to the order and index in the metadata, restore the reassembled data to the original data structure according to the structural information in the metadata, and restore the order of the structured time-series data according to timestamps or other identifiers.

[0087] Step 3: Data storage.

[0088] The process of saving decompressed and reconstructed data to a pre-specified database or data storage system. In this step, the reconstructed data is persistently stored to ensure that subsequent applications and processes can correctly read and use it. The goal of the data storage process is to ensure the long-term availability, consistency, and integrity of data, thereby supporting the normal operation of the system and the fulfillment of business needs.

[0089] (4) Optimal total duration:

[0090] To ensure data integrity and accuracy, thereby guaranteeing the credibility and validity of information, and to transmit data to the target location as quickly as possible to support timely feedback and application response, data must be sent to the recipient in high quality and in real time from the moment it is generated. The total time, from the start of data compression until the data is decompressed at the receiving end, is:

[0091] T = T c (M)+T e (M ′ )+T t (C)+T d (C)+T u (M′) (1)

[0092] Among them, the data volume and speed of each step are as follows: Figure 5 As shown.

[0093] Figure 5 The area of ​​the boxes in the graph represents the data size, while the bars represent the processing time for compression / decompression, encryption / decryption, and data transmission. As can be seen from the formula and graph, these times are not independent of each other. Using a compression algorithm or compression level with a high compression ratio results in a higher compression ratio, smaller compressed data size, and correspondingly shorter encryption and transmission times. However, the compression time will be higher because the compression algorithm performs more rounds and deeper pattern discovery and aggregation, leading to higher computational complexity. Conversely, lower compression times result in smaller compressed data sizes, but longer encryption and transmission times.

[0094] Furthermore, the time overhead coefficients for transmitting the same amount of data are different for compression, decompression, encryption, and decryption. However, compression and decompression are approximately equal, as are encryption and decryption with the same key length. At different encryption levels, higher levels require longer encryption and decryption times. Transmission time overhead is the largest, especially in satellite network environments where long signal transmission paths significantly increase transmission latency. Moreover, transmission time overhead is not fixed but varies with network conditions (such as bandwidth, latency, and packet loss rate). Real-time transmission rate needs to be determined through network feedback. By monitoring network performance indicators in real time, the current transmission rate can be dynamically evaluated, and transmission strategies can be adjusted accordingly to optimize the total data transmission time.

[0095] In summary, the relationships between different time points are intertwined, requiring comprehensive consideration of various factors to achieve overall optimization. This embodiment of the invention uses nonlinear programming to find the optimal total duration. The constraints and relationships are as follows:

[0096] The relationship between compression and transmission: Transmission time T t With compression time T C Inversely proportional, can be expressed by a constant k t To describe this relationship, let B represent the fluctuation in transmission time due to bandwidth:

[0097]

[0098] The relationship between compression and encryption: Encryption time T e With compression time T C Inversely proportional, can be expressed by a constant k e Let E be the number associated with the encryption level.

[0099]

[0100] Relationship between compression and decompression: Compression time T C With decompression time T u Proportional, can be expressed by a constant k u To describe:

[0101] T u =k u ×T C (4)

[0102] Relationship between decryption and compression: Decryption time T d With compression time T c Inversely proportional, can be expressed by a constant k d To describe:

[0103]

[0104] Substituting formulas (2)-(5) into formula (1) yields the final expression for the total time T:

[0105]

[0106] According to formula (6), nonlinear programming optimization is performed, and the minimum function is called to obtain the optimal total time.

[0107] The minimize function uses a numerical optimization algorithm to iteratively search for the minimum value of the objective function under initial guesses and constraints. The optimization steps of the function are as follows:

[0108] Step 1: Calculate the gradient.

[0109] By calculating the objective function relative to variable T C The first derivative (gradient) of the objective function is used by optimization algorithms to determine the search direction. The gradient represents the rate of change of the objective function at a specific point; moving in the direction of gradient descent decreases the value of the objective function. In each iteration, the algorithm calculates the gradient at the current point and uses this gradient information to update T. C The value of is thus gradually reduced, thereby decreasing the value of the objective function.

[0110] Step 2: Iterative update.

[0111] Iterative update is a repetitive process in which the algorithm calculates the gradient and updates T at each step. C The value of is used to gradually bring the objective function closer to its minimum. This process continues until the value of T that minimizes the objective function is found. C .

[0112] Step 3: Convergence determination.

[0113] Convergence determination refers to the process during optimization where the algorithm stops searching when the objective function value changes very little or reaches a preset convergence condition. When the change in the objective function value falls below a certain threshold or the maximum number of iterations is reached, the algorithm considers itself to have found the optimal solution and stops iterating. This mechanism ensures the algorithm completes within a reasonable timeframe and avoids unlimited iteration.

[0114] This invention also provides an efficient data compression and encryption / decryption device under low bandwidth conditions, employing the aforementioned efficient data compression and encryption / decryption method under low bandwidth conditions, comprising: a total duration optimization module, an efficient data compression module based on data classification, a data encryption module based on a predefined security policy, a data decryption module, and a data decompression and restoration module.

[0115] The total duration optimization module uses an optimization method based on the current network performance to obtain the compression and encryption scheme with the shortest total duration for compression, encryption / decryption, and transmission, and determines parameters such as compression method, compression level, and encryption level.

[0116] When planning a compression and encryption scheme, follow these steps:

[0117] a. Monitor network performance metrics in real time, including bandwidth, latency, packet loss rate, and other information.

[0118] b. Dynamically assess the current network transmission speed and estimate the transmission time required to transmit a certain amount of data under the current network conditions.

[0119] c. Using nonlinear programming, with the total delay of the entire process of data compression, encryption, transmission, decryption and decompression as the optimization index, the minimum security index and distortion rate as constraints, and parameters such as compression algorithm type, compression level and encryption level as optimization objects, the optimization problem is solved to obtain the compression and decryption scheme with the shortest total delay while satisfying the constraints.

[0120] The following steps are followed using a nonlinear programming method:

[0121] c1. Calculate the gradient: Calculate the first derivative (gradient) of the objective function with respect to the variable to determine the search direction.

[0122] c2, Iterative Update: Repeat c1 and update the total latency value. During this process, the total latency gradually decreases until the compression and decryption parameters that minimize the total latency are found.

[0123] c3. Convergence determination: After each iteration, check whether the number of iterations, optimization time, objective function value, etc. exceed the set threshold. If the threshold is reached, stop the iteration to ensure that the optimization algorithm is completed within a reasonable time and avoid unlimited iteration.

[0124] The efficient data compression module based on data classification classifies the data to be compressed according to its data type and applies the optimal compression algorithm accordingly, thereby improving compression efficiency and generating compressed data packets.

[0125] To compress the raw data, follow these steps:

[0126] (1) Data classification: The original data is parsed and first divided into structured time series data and text data. The structured time series data is further classified into integer data, floating-point data and Boolean data, etc.

[0127] (2) Selecting a compression algorithm: Select the optimal compression algorithm for different types of data. For example, use the Delta-of-Delta algorithm and Zig-Zag encoding for integer data, and use the ZFP algorithm for floating-point data.

[0128] (3) Data aggregation: Aggregate the various compressed data into a compressed package and record information such as compression method and data type, original size, and index for each data to form metadata.

[0129] The data encryption module based on a predefined security policy selects the encryption level and encryption method according to the type and degree of the data to be encrypted and the predefined encryption security policy, and encrypts the compressed data packet to generate an encrypted data packet.

[0130] Data encryption follows these steps:

[0131] A. Security Level Setting: Based on predefined security policies, the importance of the data is assessed to determine the encryption level of the data.

[0132] B. Data Encryption: Select the corresponding encryption algorithm for encryption based on the encryption level.

[0133] The data decryption module decrypts the encrypted data packets according to the corresponding encryption algorithm, restoring the data packets to compressed data packets.

[0134] The data decompression and restoration module decompresses the compressed data packet according to the corresponding compression algorithm, and then reassembles and restores the decompressed data files separated by data type to obtain the original data.

[0135] Data decompression and restoration are performed according to the following steps:

[0136] s1. Compression Algorithm Identification: Based on the compression method recorded in the metadata of the compressed data packet, determine the decompression algorithm corresponding to each type of data.

[0137] s2. Data decompression: Decompress each compressed file in the compressed data package according to the corresponding decompression algorithm to obtain several data files classified by data type.

[0138] s3. Extract index information from metadata, restore it according to the corresponding structure and time sequence information, and restore several decompressed data files to the original files.

[0139] Key technical aspects of this invention include selecting the most suitable compression algorithm based on data type, employing different encryption levels according to data importance, and optimizing total transmission time through nonlinear programming methods. These innovations effectively improve data transmission efficiency and security, and are particularly suitable for data transmission needs under low bandwidth conditions.

[0140] This invention employs various data compression algorithms to improve data transmission efficiency and security. These algorithms collectively constitute the core technology of this invention, and the algorithms used in this invention will be described in detail below:

[0141] (1) Delta of Delta:

[0142] Delta of Delta, also known as second-order differential coding, is a method that uses Delta coding again on top of Delta coding. It is more suitable for encoding monotonically increasing or decreasing sequences of data. For example, 2,4,4,6,8 becomes 2,2,0,2,2 after Delta coding, and then becomes 2,0,-2,0,0 after further Delta coding.

[0143] Facebook's Gorilla uses delta-of-delta encoding to provide a relatively general processing scheme for data across different time spans (a + b = c, where a represents the storage of the identifier bit and b represents the number of bits required to store the data), as shown in Table 1. The compression capability of delta-of-delta encoding is shown in Table 2.

[0144] Table 1 General Processing Scheme Table

[0145] Data range Flags Total bits occupied 0 0 1 [-63，64] 10 2+7＝9 [-255，256] 110 3+9＝12 [-2047，2048] 1110 4+12＝16 >2048 1111 4+32＝36

[0146] Table 2. Compression Capabilities of Delta-of-Delta Encoding

[0147]

[0148]

[0149] Assuming the start timestamp is 1571889600000, and the maximum range threshold of delta is 3600s, the storage space usage is compared as follows:

[0150] The Delta-of-delta algorithm: 64 + 9*4 + 1*3 = 103 bits. It can be seen that the Delta-of-delta algorithm achieves a higher compression ratio than the delta algorithm. In practical applications, the timestamps of massive time-series data are dense and continuous, and most of them satisfy the condition delta-of-delta = 0, which can significantly reduce the storage space of the timestamps.

[0151] Compared to the uncompressed raw timestamp data generated within 3600 seconds, if one data point is generated every 3ms, the total number of bits is: 3600*1000 / 3*64=745kbit, and the compression rate is approximately 1 / 7000.

[0152] (2) Zig-Zag algorithm:

[0153] The first sign bit of the two's complement is placed at the end of the two's complement, and the other bits are shifted one bit to the front. For negative numbers, all data bits are inverted, and the sign bit remains unchanged. For non-negative integers, the sign bit is moved to the end, and the other bits are shifted one bit to the front, while the data remains unchanged.

[0154] (3) ZFP algorithm:

[0155] The steps of the ZFP compression algorithm are as follows:

[0156] Step 1: The d-dimensional array is divided into several parts of size 4. d A d-dimensional array is called a block. If the d-dimensional array cannot be precisely divided into blocks, the boundaries of the d-dimensional array are padded until it can be precisely divided.

[0157] Step 2: Convert floating-point values ​​in each block to block floating-point representation: Convert all numbers in the block to signed binary format, and extract the exponent of the data with the largest absolute value in the block as the common exponent. In this way, all floating-point numbers in the block are represented by binary integers and a common exponent.

[0158] Step 3: Perform high-performance customized decorrelation on integers using a method similar to Discrete Cosine Transform (DCT). This approach, like DCT, reduces data correlation through a series of mathematical operations on the transformation matrix. The reason for decorrelation is that continuous fields often exhibit autocorrelation, which can be considered redundant information. By decorrelating, the data becomes more dispersed after the transformation, allowing for the representation of the same or similar information with fewer bits, thus improving compression efficiency.

[0159] Step 4: Sort the coefficients in descending order of magnitude: Coefficient size is often inversely related to the sequence. The purpose of sequence sorting is to arrange the coefficients roughly in descending order of magnitude, thus separating 1s and 0s on each bit plane. This is because coefficients at the beginning are usually larger non-zero coefficients, while smaller coefficients at the end are often zero or close to zero. Through this step, most of the smaller coefficients will be placed at the end of the sequence, and small coefficients usually share the same number of leading zeros, enhancing zero clustering. Leading zeros can be encoded into a more compact form, thereby reducing the amount of compressed data.

[0160] Step 5: Convert the signed binary integer in two's complement to a negative binary: The sign bit is usually the leftmost bit in any traditional binary representation, and it doesn't provide any useful information until it encounters the leading bit (i.e., the transition from 0 to 1 (positive value) or from 1 to 0 (negative value)). However, the first "1" in a negative binary representation reveals the sign and value range of the entire number. For example, if the leftmost bit of a negative binary number is at position 'e', ​​then the number's value is 2^32. e • Within the range [1 / 3, 4 / 3]. If e is even, then the number is positive; otherwise, it is negative. Therefore, a two's complement signed integer (standard integer representation) is converted to a negative binary representation. Negative binary representation also concentrates errors introduced by calculation or processing near zero rather than spreading them across the entire numerical range.

[0161] Step Six: For 4 d The bits of the integer list are transposed to sort them by bit plane (from most significant bit to least significant bit) instead of by coefficient.

[0162] Step 7: Lossless compression of coefficients using embedded encoding: This encoding method addresses the characteristic of numerous leading zeros in transform coefficients by dividing each bit plane into two parts: First, n least significant bits are directly output without encoding, where n is dynamic and determined by the number of encoded bits, ensuring that the high 4 bits of all previous bit planes are compressed. d - n bits are all zeros, and n is set to 0 during initialization; then, for the remaining bits, unary run-length encoding is used. Finally, the compressed code for each bit plane is output.

[0163] Step 8: Output Compressed Data: Through the preceding operations of dividing, transforming, sorting, and embedding the input data, an encoded bitstream is finally generated. The generated data is then sent bit by bit using an embedded encoder until a stop criterion is met. An embedded encoder is used in this step because it employs an adaptive encoding method that can more flexibly adjust the encoding based on the previously encoded bits, minimizing the length of the compressed bitstream. Bit-by-bit transmission improves accuracy and compression efficiency while maintaining truncation flexibility and adapting to the distribution of transform coefficients. The specific stop criterion depends on the ZFP compression mode: fixed rate, fixed precision, or fixed accuracy.

[0164] (4) Bit-Packing algorithm:

[0165] In most programming languages, Boolean values ​​typically occupy 8 bits, with 0 representing false and 1 (or greater than 1) representing true. Bit-packing represents Boolean arrays as byte arrays, and the steps are as follows:

[0166] ① Byte index: floor(i / 8);

[0167] ②bit index: i mod 8;

[0168] ③ Bit shift operation, followed by merging. For example: a boolean array containing 4 boolean values: 00000001 00000010 00000011 00000000

[0169] After shifting and merging, we get: 00111001

[0170] Decoding simply requires performing the reverse operation.

[0171] The example above uses one byte to represent the encoded result. For longer boolean arrays, multiple bytes, i.e., a byte array, are needed to represent them.

[0172] Compression ratio analysis: For Boolean data, it occupies one byte (8 bits), but only the last bit of 0 or 1 is actually valid. Therefore, when compressing Boolean data, the compression ratio using Bit-Packing is very high: 8 Boolean values ​​originally occupied 8 bytes (64 bits), but after using Bit-Packing compression, only 1 byte (8 bits) is needed, with a compression ratio as high as 12.5%.

[0173] (5) LZW algorithm:

[0174] The LZW compression algorithm achieves compression by constructing a dictionary. First, individual characters are added to an initial dictionary. Then, the input data is scanned, and the dictionary is built incrementally, replacing already encountered strings with their indices. During the scan, when a string not present in the dictionary is encountered, it is added and assigned a new index. This process is repeated until the scan is complete. By outputting the sequence of indices of the strings in the dictionary, the LZW algorithm can perform lossless compression of the input data and achieve excellent compression results.

[0175] This invention comprises two parts: a sender and a receiver. The sender includes data generation, compression, encryption, and transmission. The data compression part can select different compression algorithms based on different data types, and the data encryption part can select different encryption levels based on the importance of the data. The receiver includes receiving data, decrypting, decompressing, and storing it. To obtain the optimal total transmission time, a nonlinear programming method is used to optimize the compression time, encryption time, transmission time, decryption time, and decompression time.

[0176] In summary, the main beneficial effects and contributions of the embodiments of the present invention can be summarized as follows:

[0177] (1) When compressing data, select the most suitable compression algorithm according to the type of data, thereby improving compression efficiency and compression rate, significantly reducing data volume, saving bandwidth, and optimizing transmission performance.

[0178] (2) When encrypting data, different encryption levels are selected according to the importance of the data, so as to reduce the time and computing resources consumed in encryption and decryption while ensuring necessary security, thereby improving the security and efficiency of the system.

[0179] (3) By using nonlinear programming methods, the time relationships of each stage of data compression, encryption, transmission, decryption, and decompression are optimized to obtain the optimal total duration under variable conditions such as compression algorithm selection, compression level, encryption level, and transmission status. This ensures that data can be transmitted efficiently, quickly, and securely under narrow bandwidth conditions, thereby improving the overall efficiency and performance of data transmission.

Claims

1. A high-efficiency data compression and encryption / decryption method under low bandwidth conditions, characterized in that, Includes the following steps: The sender acquires various types of environmental data in real time, classifying the environmental data into structured time-series data and text data, and further classifying the structured time-series data into integer data, floating-point data, and Boolean data. For integer data, the Delta-of-Delta algorithm and Zig-Zag encoding are used for compression; for floating-point data, the ZFP algorithm is used for compression; for Boolean data, the Bit-Packing algorithm is used for compression; and for text data, the LZW algorithm is used for compression. Based on the data source, content, and importance of the content in a specific application scenario, a security policy is predefined. The importance of the compressed integer data and text data is evaluated according to the predefined security policy to obtain the corresponding security level of the compressed integer data and text data. An encryption algorithm is used in combination with the security level to segment and encrypt the compressed integer data and text data, and the encrypted data stream is used as the output ciphertext. Real-time monitoring of data transmission network performance indicators, calculation of the actual transmission speed of the current data transmission network or the transmission time required for a certain amount of data and dynamic evaluation, and adjustment of transmission strategy based on dynamic evaluation results to transmit the output ciphertext to the receiver. The receiver obtains the output ciphertext and matches it with the corresponding decryption algorithm to segment and decrypt it, thus obtaining the decrypted data stream. Based on the compression algorithm corresponding to the decrypted data stream transmitted by the sender, if the compression algorithm is Delta-of-Delta and Zig-Zag encoding, then the integer data is restored by reverse calculation of the data difference sequence and encoding conversion; if the compression algorithm is ZFP, then the floating-point data is gradually restored by reverse prediction encoding and quantization operations; if the compression algorithm is Bit-Packing, then the Boolean data is restored by unpacking the bitmap data; if the compression algorithm is LZW, then the text data is restored by constructing a reverse dictionary to restore the compressed encoding sequence. Based on the original data order, data index information, and structure information corresponding to the decrypted data stream transmitted by the sender, the decompressed data stream is restored to its original data structure, and the order of the structured time-series data is restored according to the timestamp to obtain the recombined data. The decompressed data stream and the reassembled data are stored.

2. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 1, characterized in that, When the sender performs the compression, it sets the corresponding compression algorithm parameters according to the selected algorithm.

3. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 1, characterized in that, The encryption algorithm used is AES.

4. The efficient data compression and encryption method under low bandwidth conditions as described in claim 1, characterized in that, The segmentation and encryption process includes the following steps: The compressed integer and text data are divided into data blocks of a preset size; Convert the original key into a series of subkeys; Perform an XOR operation between the data block and the subkey of the initial round to achieve initial round encryption; The byte substitution, row shift, column obfuscation, and round key addition operations are performed in predetermined rounds until the final round of encryption, which performs byte substitution, row shift, and round key addition operations to obtain an encrypted data stream as the output ciphertext.

5. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 1, characterized in that, The segmentation and decryption process includes the following steps: The output ciphertext is divided into data blocks of a preset size; Perform an XOR operation between the data block and the subkey of the initial round to achieve initial round decryption; The process involves performing reverse byte substitution, reverse row shift, reverse column obfuscation, and round key addition operations in predetermined rounds until the final round of encryption performs reverse byte substitution and reverse row shift operations to obtain the decrypted data stream.

6. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 1, characterized in that, The efficient data compression and encryption / decryption method includes: The optimal total duration min(T) is calculated using the formula for total duration T: T=T c (M)+T e (M ′ )+T t (C)+T d (C)+T u (M′) Where M is structured time-series data, M′ is compressed structured time-series data that meets the minimum distortion rate requirement, and M′ is also the decrypted data stream after decryption by the receiver, C is encrypted data stream, and T... c The compression time of M, T e It is the encryption time of M', T t T is the transmission time of C. d It is the decryption time of C, T u This is the decompression time of M′; The timer for the sender to compress the structured time-series data is calculated based on the optimal total duration min(T) until the receiver decompresses the decrypted data stream to obtain the decompressed decrypted data stream. The transmission strategy is adjusted based on the optimal total duration.

7. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 6, characterized in that, The calculation of the optimal total duration min(T) according to the formula for total duration T includes: Based on the formulas for the relationship between compression and transmission, compression and encryption, compression and decompression, and decryption and compression, combined with the formula for total duration T, the final expression for total duration T is obtained. The optimal total duration min(T) is obtained by performing nonlinear programming optimization on the final expression of the total duration T by calling the minimize function.

8. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 7, characterized in that, The formula relating compression and transmission is as follows: Where B represents the fluctuation in transmission time due to bandwidth, and k t Represents compression and transmission constants; The formula relating compression and decompression is as follows: T u =k u ×T C Where, k u This represents the compression and decompression constants.

9. The efficient data compression and encryption method under low bandwidth conditions as described in claim 7, characterized in that, The formula relating compression and encryption is as follows: Where E represents the encryption level, k e Represents compression and encryption constants; The formula relating decryption and compression is as follows: Where, k d This represents the decryption and compression constants.

10. The efficient data compression and encryption / decryption method under low bandwidth conditions as described in claim 7, characterized in that, The optimization steps in calling the minimize function are as follows: The gradient of the final expression of the total duration T relative to the compression time of the structured time series data is calculated. The value of the final expression of the total duration T is updated and iterated by moving along the direction of gradient descent to reduce the value of the final expression of the total duration T. It is then determined whether the change in the compression time of the structured time series data reaches a preset threshold. If it does, the minimum value of the compression time of the structured time series data is determined.

Images