A method of data compression and packet encryption that can be performed in parallel

Abstract

The application discloses a data compression and grouping encryption method capable of parallel execution and relates to the field of data compression and encryption. The method divides to-be-processed data into multiple data groups and compression windows, predicts a compression output length according to data characteristics of the compression windows, and determines an expected encryption grouping quantity and an encryption count interval according to the compression output length; generates a data group key and a candidate key stream generation parameter, and performs parallel compression processing and candidate key stream generation; before the data group completes compression, forms encrypted subgroups from the generated variable-length compressed data and performs encryption processing based on a grouping cipher; expands or invalidates the candidate key stream according to an actual compression output length, and sequentially packages the encrypted compressed data according to an encrypted subgroup index. The method can reduce waiting time between compression and encryption, improve processing efficiency, and avoid repeated use of an encryption count position.

Description

Technical Field

[0001] This invention relates to the field of data compression and encryption, and in particular to a data compression and block encryption method that can be executed in parallel. Background Technology

[0002] With the ever-increasing scale of data storage, network transmission, and cloud processing, data to be processed typically needs to be compressed and encrypted sequentially to reduce data transmission volume and ensure data security. Existing technologies generally compress the entire data segment or block first, then encrypt the compressed data. The compression and encryption processes have a clear dependency, causing the encryption unit to wait until compression is complete, making it difficult to fully utilize the parallel processing capabilities of multi-core processors, dedicated compression units, and cryptographic operation units. While some existing solutions divide the data to be processed into multiple blocks and perform compression and encryption on different blocks separately, this primarily creates a pipelined processing flow between different blocks, failing to address the issue of overlapping compression and encryption processes within the same data group.

[0003] Furthermore, compressed data generated during compression processing typically has a variable output length. Different compression windows exhibit varying data entropy, duplicate data ratios, and compression dictionary matching, making it difficult to accurately determine the required number of encryption blocks and keystream length before compression is complete. Generating encryption parameters only after all compressed data has been processed increases overall processing latency; pre-generating the keystream according to the original data length can easily lead to insufficient keystream or a large surplus. Simultaneously, when compression and encryption processes are executed in parallel, issues such as compressed data buffer backlog, reuse of encryption count intervals, out-of-order completion of different encryption subgroups, and difficulty in synchronously restoring the compression and encryption states after abnormal interruptions must also be addressed.

[0004] To address this, a parallel-executable data compression and block encryption method is proposed. Summary of the Invention

[0005] The main objective of this invention is to provide a data compression and block encryption method that can be executed in parallel, which can effectively solve the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A data compression and block encryption method that can be executed in parallel includes the following steps:

[0008] S1. Obtain the data to be processed, divide the data to be processed into multiple sequentially arranged data groups, and divide each data group into multiple compressed windows, and configure data group identifiers and window identifiers for each data group and each compressed window.

[0009] S2. Extract the data features of the current compression window, predict the compression output length of the current compression window based on the data features, and determine the expected number of encrypted packets and the encryption count interval corresponding to the expected number of encrypted packets based on the predicted compression output length.

[0010] S3. Generate the data group key corresponding to the current data group based on the master key, processing task identifier, data group identifier of the current data group and session random number, and determine the candidate key stream generation parameters based on the data group key, the window identifier of the current compression window and the encryption count interval.

[0011] S4. Start the compression processing task and the candidate key stream generation task of the current data group in parallel. The compression processing task reads the original data in the current data group in sequence, writes the generated variable-length compressed data into the compression output buffer continuously, and generates a candidate key stream according to the candidate key stream generation parameters through the candidate key stream generation task, and writes the candidate key stream into the key stream buffer.

[0012] S5. Based on the compression write position and encryption read position in the compression output buffer, continuously determine the unencrypted compressed data corresponding to the current compression window; before all the original data of the current data group is compressed, when the unencrypted compressed data reaches the preset encryption group length, extract compressed data from it to form an encrypted subgroup; when all the original data of the current data group is compressed and there is remaining unencrypted compressed data that is less than the preset encryption group length, form the remaining unencrypted compressed data into a tail encrypted subgroup; configure the encrypted subgroup sequence number, compression start offset, and effective data length for the encrypted subgroup and the tail encrypted subgroup respectively;

[0013] S6. Determine the corresponding encryption count position according to the compression start offset of each encryption subgroup. Before all the original data of the current data group is compressed, start the encryption process for the formed encryption subgroups. Use the data group key and the candidate key stream of the corresponding encryption count position to perform block cipher-based encryption process on each formed and subsequently formed encryption subgroup to obtain encrypted compressed data and authentication tags.

[0014] S7. Obtain the actual compressed output length of the current compression window, and adjust the candidate key stream according to the difference between the actual compressed output length and the predicted compressed output length. When the candidate key stream is insufficient, allocate an extended counting interval and generate an extended key stream. When there is a remaining candidate key stream, mark the encryption counting position corresponding to the unused candidate key stream as invalid.

[0015] S8. Establish an encryption subgroup index based on the data group identifier, window identifier, encryption subgroup sequence number and compression start offset of each encryption subgroup, and encapsulate each encrypted compressed data in an orderly manner according to the encryption subgroup index to generate the encryption and compression result corresponding to the data to be processed.

[0016] Furthermore, the data features include at least the normalized data entropy of the current compression window, the compression dictionary matching rate, the proportion of literal data, and the actual compression rate of the previous compression window; the predicted compression rate of the current compression window is determined according to the following formula:

[0017]

[0018] in, For the first The predicted compression ratio for each compression window. This represents the actual compression ratio of the previous compression window. For the first Normalized data entropy of a compressed window, For the first The matching rate of the compressed dictionary for each compressed window. For the first The proportion of literal data in each compressed window. , , and The weights are preset and the sum of the weights is 1. and These are the preset lower compression ratio limit and the preset upper compression ratio limit, respectively. This means that the calculation results will be limited to between the preset lower limit and the preset upper limit of compression ratio;

[0019] For the first compression window in the current data group, the actual compression ratio of the previous compression window is taken as the preset initial compression ratio.

[0020] Furthermore, based on the difference between the predicted compression ratio and the actual compression ratio of the compression windows preceding the current compression window, the prediction error of the current compression window is determined; the predicted compression output length of the current compression window is determined according to the following formula:

[0021]

[0022] in, For the first The predicted compressed output length of each compression window. For the first The original data length of each compressed window. For the first The prediction error of each compressed window, This is the preset error amplification factor. To preset the encrypted packet length, This indicates rounding up; the expected number of encrypted packets is determined by rounding up the ratio of the predicted compressed output length to the preset encrypted packet length; for the first compression window in the current data group, the prediction error is taken as the preset initial prediction error.

[0023] Furthermore, non-overlapping encryption count intervals are allocated to each compression window using an atomic counting cursor. The atomic counting cursor value before allocation is used as the starting count value corresponding to the current compression window, and the atomic counting cursor is updated according to the expected number of encryption blocks after allocation is completed. The expected number of encryption blocks in the current compression window and the starting encryption count position corresponding to the j-th encryption subgroup are determined according to the following formula:

[0024] ,

[0025] in, For the first The expected number of encrypted packets per compression window. For the first The predicted compressed output length of each compression window. To preset the encrypted packet length, For the first The first compression window The starting encryption count position corresponding to each encrypted subgroup For the first The starting count value corresponding to each compression window For the first The encrypted subgroup in the th _ The compression start offset in the compression output corresponding to each compression window. This indicates rounding down to the nearest integer.

[0026] Furthermore, the compressed output buffer is a circular buffer, which is configured with a compressed write pointer, an encrypted read pointer, and a committed pointer. The buffer occupancy ratio is determined based on the amount of data between the compressed write pointer and the encrypted read pointer. When the buffer occupancy ratio is greater than a preset high occupancy threshold, an encryption priority processing signal is generated by the scheduling controller and sent to the task queue selection interface, so that the encryption processing unit prioritizes reading the encrypted subgroup to be encrypted in the next scheduling cycle, or sends a compression pause signal to the compression processing unit to pause the entry of a new compression window into compression processing. When the buffer occupancy ratio is less than a preset low occupancy threshold, a compression priority processing signal is generated by the scheduling controller and sent to the task queue selection interface, so that the compression processing unit prioritizes reading the compression window to be compressed in the next scheduling cycle, or releases the compression pause signal.

[0027] Furthermore, the encrypted subgroup is subjected to block cipher-based encryption processing, including: determining the starting encryption count position corresponding to the encrypted subgroup based on the compression start offset and the preset encryption block length; setting at least one of the data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, and compression parameter information as additional authentication data; and using an authentication encryption mode with a counter structure, encrypting the encrypted subgroup based on the data group key, the starting encryption count position, and the additional authentication data to generate the encrypted compressed data and authentication tag; wherein different encrypted subgroups within the same data group correspond to different encryption count positions.

[0028] Furthermore, it also includes generating joint state checkpoints at preset checkpoint intervals. Each joint state checkpoint records at least the data group identifier, the position of the original data read, the window identifier of the current compression window, compression status information, compression write position, encryption read position, the sequence number of the submitted encryption subgroup, the atomic count cursor, and the length of the completed encryption data. When an anomaly occurs in the compression or encryption task, the most recently verified joint state checkpoint is restored, and the compressed data, encrypted compressed data, and corresponding candidate key streams that have not yet been confirmed and submitted after the joint state checkpoint are deleted. The allocated encryption count intervals after the joint state checkpoint are marked as invalid. The maximum end count position in the allocated encryption count intervals is determined, the atomic count cursor is updated to the count position after the maximum end count position, and the encryption count intervals and candidate key streams are reassigned from the updated atomic count cursor to continue executing the compression and encryption tasks.

[0029] Furthermore, the encrypted subgroup index records at least the data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, encryption count position, authentication tag, and processing status of each encrypted subgroup; it allows multiple encrypted subgroups belonging to the same data group to complete encryption processing in an order different from the encrypted subgroup sequence number, and sorts and encapsulates the encrypted compressed data that has completed encryption processing according to the window identifier, encrypted subgroup sequence number, and compression start offset; and it generates group-level authentication information based on the encrypted subgroup sequence number, compression start offset, effective data length, and authentication tag of each encrypted subgroup after sorting.

[0030] A parallel-executable data compression and block encryption system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the following functional modules:

[0031] The data partitioning module is used to acquire the data to be processed, divide the data to be processed into multiple sequentially arranged data groups, and divide each data group into multiple compressed windows;

[0032] The compression output prediction module is used to predict the compression output length based on the data characteristics of the current compression window, and to determine the expected number of encrypted packets and the encryption count interval based on the predicted compression output length.

[0033] The key stream parameter determination module is used to generate the data group key corresponding to the current data group, and determine the candidate key stream generation parameters based on the data group key, window identifier and encryption count interval;

[0034] The parallel processing module is used to start compression processing tasks and candidate key stream generation tasks in parallel, write the generated variable-length compressed data into the compression output buffer, and write the generated candidate key stream into the key stream buffer.

[0035] The encrypted subgroup generation module is used to form encrypted subgroups based on the unencrypted compressed data in the compressed output buffer before all the original data of the current data group is compressed.

[0036] The block encryption module is used to determine the corresponding encryption count position based on the compression start offset of the encrypted subgroup before all the original data of the current data group is compressed, and to perform block cipher-based encryption processing on the encrypted subgroup using the data group key and the candidate key stream corresponding to the encryption count position.

[0037] The keystream adjustment module is used to generate an extended keystream based on the difference between the actual compressed output length and the predicted compressed output length, or to mark the encryption count position corresponding to the unused candidate keystream as invalid.

[0038] The result encapsulation module is used to establish an encrypted subgroup index based on the data group identifier, window identifier, encrypted subgroup sequence number and compression start offset of each encrypted subgroup, and to encapsulate each encrypted compressed data in an orderly manner according to the encrypted subgroup index.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method steps.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention starts compression processing and candidate key stream generation tasks in parallel for the same data group, and starts encryption processing on the already formed encrypted subgroups before all the original data in the current data group is compressed. This makes compression processing, key stream generation and group encryption overlap in time, reduces the time the encryption processing unit waits for compression to complete, and improves the overall processing efficiency of data compression and encryption.

[0042] 2. This invention predicts the compressed output length based on the data characteristics of the compressed window, and accordingly determines the expected number of encrypted blocks and the encryption count interval. At the same time, it expands or invalidates the candidate key stream according to the actual compressed output length, so that the pre-generated key stream can adapt to the situation of uncertain compressed data length, avoids the interruption of encryption processing due to insufficient key stream, and reduces the security risk of unused key streams being repeatedly distributed.

[0043] 3. This invention dynamically forms encrypted subgroups based on the compressed write position and encrypted read position in the compressed output buffer, and establishes an encrypted subgroup index through the encrypted subgroup sequence number, compression start offset and effective data length, so that multiple encrypted subgroups can complete the encryption process in parallel and out of order, while still being encapsulated in an ordered manner according to the original compressed data order, thus taking into account both parallel processing efficiency and data recovery accuracy.

[0044] 4. By setting joint state checkpoints, this invention records the compression state, cache position, submitted encrypted subgroups, and encryption count position in association. When an anomaly occurs during compression or encryption, the encryption count interval allocated after the checkpoint is marked as invalid, and the processing is re-executed from the new encryption count position. This avoids the reuse of candidate key streams during anomaly recovery and improves the continuity and security of parallel compression and block encryption processes. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0046] Figure 2 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Example 1

[0049] like Figure 1-2 As shown, a data compression and block encryption method that can be executed in parallel can be deployed in servers, cloud storage nodes, network transmission equipment, edge computing devices, or terminal devices with compression and cryptographic capabilities. The execution device may include a processor, memory, compression processing unit, encryption processing unit, scheduling controller, compression output buffer, keystream buffer, and result storage area.

[0050] The compression processing unit and the encryption processing unit can be implemented by different processing cores of the same processor, or they can be implemented by a central processing unit, a graphics processing unit, a field-programmable gate array, or a dedicated compression and encryption accelerator, respectively.

[0051] Parallel execution in this invention refers to initiating block encryption of the compressed data generated earlier in a data group before the entire data group has been compressed, while simultaneously generating candidate key streams required for subsequent encryption subgroups. It does not require the same original data byte to be compressed and encrypted simultaneously at the same time.

[0052] At a specific processing moment, the compression processing unit can compress the original data at the end of the current data group, the encryption processing unit can encrypt the compressed data that has been generated at the beginning of the current data group, and the candidate key stream generation task can simultaneously generate the candidate key stream required for subsequent encryption subgroups.

[0053] Data partitioning

[0054] Obtain the data to be processed and divide it into multiple sequentially arranged data groups according to the preset data group length.

[0055] The data to be processed can be file data, database backup data, network data stream, image data, video data, device log data, or sensor-collected data.

[0056] For example, if the total length of the data to be processed is 40MB and the preset data group length is 4MB, then the data to be processed is divided into 10 data groups. Each data group is assigned a data group identifier G according to its order within the data to be processed.

[0057] Each data group is further divided into multiple compression windows. For example, each 4MB data group is divided into 64 compression windows of 64KB each, and a window identifier is configured for each compression window. .

[0058] Among them, the data group serves as the basic unit for key derivation, group-level authentication, result encapsulation, and anomaly recovery, while the compression window serves as the basic unit for data feature extraction, compression output prediction, encryption count interval allocation, and compression task scheduling.

[0059] Data located at the end of the data to be processed and whose length is less than the preset data group length is still processed as a data group; data located at the end of the data group and whose length is less than the preset compression window length is still processed as a compression window, and its actual data length is recorded.

[0060] Data feature extraction and compression output length prediction

[0061] For the current compression window, extract the normalized data entropy, compression dictionary matching rate, literal data ratio, and the actual compression rate of the previous compression window.

[0062] In one embodiment, bytes are used as data symbols, and the number of symbol classes m is 256. The normalized data entropy of the current compression window. Determined according to the following formula:

[0063]

[0064] in, For the first Normalized data entropy of a compressed window, For the first The first compression window The probability of a data type symbol appearing. When a data type symbol does not appear, the corresponding calculation term is set to 0.

[0065] The larger the normalized data entropy, the more uniform the data distribution in the current compression window, and the lower the compressibility; the smaller the normalized data entropy, the higher the data concentration, and the higher the compressibility.

[0066] Compressed dictionary matching rate The length of the data that can be matched with existing data in the compression dictionary or sliding matching window during the compression pre-scan process is determined:

[0067]

[0068] in, For the first The total length of data that can form a valid dictionary match within each compressed window. For the first The original data length of each compressed window.

[0069] Literal data ratio This represents the proportion of the original data length that needs to be encoded according to literals to the original data length of the current compression window.

[0070] The actual compression ratio of the previous compression window It is the ratio of the actual compressed output length of the previous compression window to its original data length.

[0071] The predicted compression ratio of the current compression window is determined by the following formula:

[0072]

[0073] in, For the first Predicted compression ratio for each compression window; , , and The preset weights are used, and the following conditions are met:

[0074]

[0075] and These are the preset lower compression ratio limit and the preset upper compression ratio limit, respectively. This means that the weighted calculation result will be limited to between the preset lower limit and the preset upper limit of compression ratio.

[0076] In one example , , and Take values ​​of 0.35, 0.25, 0.25, and 0.15 respectively. Take 0.05,

[0077] Take 1.10.

[0078] The above parameters can be adjusted according to the actual compression algorithm used, historical compression results, and the data type to be processed, and are not limited to the example values ​​above.

[0079] For the first compression window in the current data group, since there is no previous compression window in the same data group, the actual compression ratio of the previous compression window can be taken as the preset initial compression ratio, for example, 0.80; or the actual compression ratio of the last compression window in the previous data group can be used.

[0080] The prediction error of the current compression window is determined based on the difference between the predicted compression ratio and the actual compression ratio of the previous compression window. In one embodiment, the prediction error is updated according to the following formula:

[0081]

[0082] in, For the first The prediction error of each compressed window, This represents the prediction error of the previous compressed window. Let be the error smoothing coefficient, and satisfy:

[0083]

[0084] For the first compressed window in the current data set, the prediction error is set to the preset initial prediction error, for example, 0.05.

[0085] The predicted compressed output length of the current compression window is determined by the following formula:

[0086]

[0087] in, For the first The predicted compressed output length of each compression window. For the first The original data length of each compressed window. This is the preset error amplification factor. To preset the encrypted packet length, This indicates rounding up to the nearest integer.

[0088] in, Used to correct the predicted compressed output length based on historical prediction errors. As an additional security margin, it is used to reduce the probability of insufficient candidate key streams.

[0089] The predicted compressed output length is only used to pre-allocate the encryption count interval and prepare candidate key streams, and is not used to limit the actual compressed output length generated by the compression algorithm.

[0090] Expected number of encrypted packets and encryption count range

[0091] The expected number of encrypted packets is determined based on the ratio of the predicted compressed output length to the preset encrypted packet length.

[0092]

[0093] in, For the first The expected number of encrypted packets per compression window.

[0094] For example, preset encrypted block length Given a current compression window with a predicted compressed output length of 38KB and a value of 16KB, the expected number of encrypted blocks is... The value is 3.

[0095] This embodiment sets up an atomic counting cursor to allocate non-overlapping encrypted counting intervals for different compression windows.

[0096] When allocating an encrypted count interval for the current compression window, the atomic count cursor value before allocation is used as the starting count value for the current compression window. And after the allocation is complete, increase the atom count cursor. .

[0097] For example, if the current atom count cursor value is 100, the expected number of encrypted blocks is... If the value is 3, then the encryption count range allocated to the current compression window is 100 to 102, and the next compression window will be allocated starting from count position 103.

[0098] By reading and updating the atomic count cursor atomically, overlapping encrypted count positions can be avoided when multiple compressed windows or multiple tasks request encrypted count intervals in parallel.

[0099] No. The first compression window The starting encryption count position corresponding to each encrypted subgroup is determined according to the following formula:

[0100]

[0101] in, For the first The first compression window The starting encryption count position corresponding to each encrypted subgroup For the first The starting count value corresponding to each compression window For the first The encrypted subgroup in the th _ The compression start offset in the compression output corresponding to each compression window. This indicates rounding down to the nearest integer.

[0102] Preset encrypted packet length The length of the cipher block can be greater than the block size of the block cipher algorithm itself. For example, the block size of AES is 128 bits, while the length of the cipher subgroup in this embodiment can be 16KB. Within a cipher subgroup, a continuous internal counting sequence can be generated based on the starting cipher count position.

[0103] Data group key and candidate key stream generation parameters

[0104] Based on the master key Task processing identifier The data group identifier of the current data group and session randomness Generate the data group key for the current data group using the key derivation function:

[0105]

[0106] in, The data group key for the current data group. Derivation function for preset key, symbol " "" indicates data splicing.

[0107] The key derivation function can be a key derivation function based on hash message authentication code, or it can be other key derivation functions that meet preset security requirements.

[0108] Session randomness Regenerated at the start of each data compression and block encryption task. Session randomness, processing task identifier, and related algorithm identifier can be stored in the task header information of the encrypted compression result.

[0109] The candidate key stream generation parameters are determined based on the data group key, data group identifier, window identifier, session random variable, and assigned encryption count interval.

[0110] In one embodiment, the first The first compression window The initial parameters of each encrypted subgroup are determined according to the following formula:

[0111]

[0112] in, These are the initial parameters for the corresponding encrypted subgroup. For the preset hash function, This indicates the length of the initial parameters required by the encryption algorithm to truncate the hash result.

[0113] By adding processing task identifier, data group identifier, window identifier, session random variable, and encryption count position to the initial parameters, different initial parameters can be assigned to different processing tasks, different data groups, different compression windows, and different encryption subgroups.

[0114] Compression processing and candidate key stream generation are performed in parallel.

[0115] After determining the data group key and candidate key stream generation parameters, the compression processing task and candidate key stream generation task for the current data group are started in parallel.

[0116] The compression processing task reads the compression windows in the current data group sequentially and generates variable-length compressed data using a preset lossless compression algorithm.

[0117] Compression algorithms can be LZ series algorithms, DEFLATE algorithm, Zstandard algorithm, or other lossless compression algorithms with dictionary matching, literal encoding, and variable-length compressed output features.

[0118] The compression task does not wait for the current data group to finish compressing, but instead continuously writes the already generated compressed data into the compression output buffer.

[0119] The candidate key stream generation task generates the candidate key streams required for the expected encryption subgroups based on the data group key, window identifier, session randomness, and encryption count range, and writes the candidate key streams into the key stream buffer.

[0120] When using an authentication encryption mode with a counter structure, block cipher operations can be performed on the corresponding counter blocks in advance to generate candidate key streams; the authentication tag is generated after the actual encrypted compressed data and additional authentication data are determined.

[0121] Therefore, while the compression processing task continues to process the subsequent raw data of the current data group, the candidate key stream generation task can simultaneously prepare candidate key streams for the encrypted subgroups that have been generated or are about to be generated.

[0122] Compressed output buffer

[0123] The compressed output buffer preferably adopts a ring buffer, which is equipped with a compressed write pointer, an encrypted read pointer, and a committed pointer.

[0124] The compressed write pointer is used to indicate the end position of the compressed data that has been generated and written to the compressed output buffer.

[0125] The encrypted read pointer is used to indicate the end position of the compressed data that has been extracted from the compressed output buffer and formed into encrypted subgroups.

[0126] The committed pointer indicates the position at the end of the compressed data that has been encrypted and written to the result storage area.

[0127] Based on the data range between the compressed write pointer and the encrypted read pointer, continuously identify unencrypted compressed data that has not yet formed encrypted subgroups.

[0128] When the write or read pointer of the circular cache reaches the end of the cache, it can return to the beginning of the cache and continue moving. The actual data location can be determined by combining the number of pointer cycles, thus avoiding confusion between a full cache and an empty cache.

[0129] Formation of ciphertext and tail ciphertext

[0130] Before all the original data in the current data group is compressed, when the unencrypted compressed data in the compressed output buffer reaches the preset encrypted group length, the compressed data is extracted according to the preset encrypted group length to form an encrypted subgroup.

[0131] For example, if the preset encrypted group length is 16KB, when 20KB of unencrypted compressed data accumulates in the compressed output buffer, the first 16KB of compressed data is extracted to form an encrypted subgroup, and the remaining 4KB is kept in the compressed output buffer.

[0132] Configure the encryption subgroup number, window identifier, compression start offset, and effective data length for each encryption subgroup.

[0133] The compression start offset represents the offset of the encrypted subgroup relative to the starting position of the compressed output of the corresponding compression window. To facilitate the sorting of encrypted subgroups among different compression windows, the window start compression offset of the compression window in the compressed byte stream of the data group can also be recorded simultaneously.

[0134] When an encrypted subgroup spans two adjacent compression windows, the window identifier of the encrypted subgroup can be set to the window identifier of the compression window to which its initial compressed data belongs, and the compression start offset can be recorded according to the compression output start position of that compression window; alternatively, the compression start offset can be uniformly recorded as the offset relative to the start position of the compressed byte stream of the current data group.

[0135] In practice, it is preferable to use the data group-level compression start offset to uniformly sort the encrypted subgroups formed across windows.

[0136] After all the original data in the current data group has been compressed, if there is still unencrypted compressed data in the compressed output buffer that is less than the preset encrypted group length, then the remaining unencrypted compressed data will be formed into a tail encrypted subgroup.

[0137] The tail-end encrypted subgroup belongs to the encrypted subgroups and uses the same encrypted subgroup sequence number, window identifier, compression start offset, effective data length, encryption count position, and authentication tag recording method as the ordinary encrypted subgroup. The only difference between the tail-end encrypted subgroup and the ordinary encrypted subgroup is that its effective data length is less than the preset encrypted block length.

[0138] For example, after the current data group is compressed, if there is 6KB of unencrypted compressed data remaining in the compressed output buffer, then this 6KB of compressed data will be formed into a tail encrypted subgroup, and its effective data length will be recorded as 6KB.

[0139] Block encryption processing

[0140] Before all the original data in the current data group is compressed, encryption processing for the formed encrypted subgroup will be initiated as long as at least one encrypted subgroup has been formed.

[0141] After encryption processing begins, it continues to receive encrypted subgroups formed during the compression process. Therefore, the encryption process starts before the current data group is fully compressed.

[0142] The starting encryption count position is determined based on the compression start offset of the encrypted subgroup, and the encrypted subgroup is encrypted using a block cipher based process with the data group key and the candidate key stream corresponding to the starting encryption count position.

[0143] In one embodiment, an authentication encryption mode with a counter structure, such as AES-GCM or SM4-GCM, is used.

[0144] Set at least one of the following as additional authentication data: data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, compression algorithm identifier, and compression parameter version.

[0145] The authentication encryption mode generates encrypted compressed data and authentication tags based on the data group key, the initial parameters corresponding to the encrypted subgroup, the additional authentication data, and the compressed data in the encrypted subgroup.

[0146] By assigning different encryption subgroups within the same data group to different encryption count positions, the reuse of the same candidate key stream under the same data group key can be avoided.

[0147] For the tail encryption subgroup, encryption can be performed only on compressed data within the effective data length range, and the effective data length can be written into the additional authentication data and the encryption subgroup index.

[0148] When the authentication encryption mode supports input of arbitrary length, the trailing encrypted subgroup does not need to be padded; when a fixed length input is required, the trailing encrypted subgroup can be padded, and the padded data can be removed after decryption based on the length of the valid data.

[0149] Candidate key stream adjustment

[0150] After the current compression window completes compression, the actual compressed output length of the compression window is obtained, and the actual compressed output length is compared with the predicted compressed output length.

[0151] When the actual compressed output length is greater than the predicted compressed output length, and the allocated candidate key stream is insufficient to cover the subsequently formed encrypted subgroups, an extended counting interval is allocated using an atomic counting cursor, and an extended key stream is generated based on the extended counting interval.

[0152] The starting counting position of the extended counting interval is located after the ending counting position of the previously allocated counting interval, and does not overlap with the encrypted counting intervals already allocated to other compressed windows.

[0153] When the actual compressed output length is less than the predicted compressed output length, there may be unused encrypted count positions in the pre-allocated encrypted count interval.

[0154] For cryptographic count positions where candidate keystreams have been generated but not used by any cryptographic subgroup, they are marked as invalid. Invalid cryptographic count positions will no longer be assigned to other cryptographic subgroups, other compression windows, other data groups, or subsequent processing tasks within the current compression window.

[0155] An encrypted counting interval status table can be established to record the starting and ending counting positions, corresponding data group identifiers, corresponding window identifiers, and statuses such as unallocated, allocated, generated, used, and invalid for each encrypted counting interval.

[0156] Cache load tuning

[0157] The cache occupancy ratio is determined based on the amount of data between the compressed write pointer and the encrypted read pointer.

[0158] When the cache usage ratio is greater than the preset high usage threshold, it indicates that the speed of compressed data generation is higher than the speed of encryption processing.

[0159] The scheduling controller generates an encryption priority processing signal and sends it to the task queue selection interface, so that the encryption processing unit can read the encryption subgroup to be encrypted first in the next scheduling cycle.

[0160] When the cache occupancy ratio continues to exceed the preset high occupancy threshold, or reaches the preset limit occupancy threshold, the scheduling controller can also send a compression pause signal to the compression processing unit to pause new compression windows from entering the compression process, but allow the currently executing compression window to complete compression.

[0161] When the cache occupancy rate is less than the preset low occupancy threshold, the scheduling controller generates a compression priority processing signal, which enables the compression processing unit to read the compression window to be compressed first in the next scheduling cycle, or to release the previously sent compression pause signal.

[0162] In one example, the preset high occupancy threshold is 70% of the compressed output buffer capacity, the preset low occupancy threshold is 30% of the compressed output buffer capacity, and the preset maximum occupancy threshold is 90% of the compressed output buffer capacity.

[0163] The above thresholds can be dynamically adjusted based on the historical processing speeds of the compression and encryption units.

[0164] Joint State Checkpoint

[0165] To achieve consistent recovery when compression and encryption processes are executed in parallel, joint state checkpoints are generated according to a preset checkpoint interval.

[0166] The joint state checkpoint records at least the data group identifier, the position of the original data read, the window identifier of the current compression window, the compression status information, the compression write position, the encryption read position, the sequence number of the submitted encryption subgroup, the atom count cursor, and the length of the encrypted data.

[0167] Compression status information may include the contents of the compression dictionary, a summary of the compression dictionary status, the sliding window status, the entropy coding status, and the compression algorithm parameters.

[0168] The joint state checkpoint can also record the keystream cache state, the encryption count interval state table version, the result storage location, and the committed pointer.

[0169] Joint state checkpoints can be triggered based on the amount of raw data processed, the number of encrypted subgroups generated, or the time interval. For example, a joint state checkpoint can be generated every 1MB of raw data processed, every 16 encrypted subgroups formed, or every 500 milliseconds.

[0170] To verify the completeness of the joint state checkpoint, a hash check value or message authentication value can be generated from the state data in the joint state checkpoint.

[0171] Abnormal recovery

[0172] When an anomaly occurs in a compression or encryption task, locate the most recently verified joint state checkpoint and restore the compression state, cache location, and committed state recorded at that joint state checkpoint.

[0173] The unconfirmed compressed data, encrypted compressed data, and corresponding candidate key streams after the deletion of the joint state checkpoint.

[0174] All encrypted count intervals allocated after this joint state checkpoint are marked as invalid, and will not be reused regardless of whether the corresponding candidate key stream has actually been used to generate encrypted compressed data.

[0175] Then, determine the maximum end count position in the allocated encrypted count interval, and update the atomic count cursor to the count position after the maximum end count position.

[0176] The updated atomic counting cursor is used to reallocate new cryptographic counting intervals and generate new candidate key streams to continue the compression and encryption tasks.

[0177] For example, if the atomic count cursor recorded at the joint state checkpoint is 200, and the encrypted count intervals allocated after this checkpoint include 200 to 203 and 204 to 206, then during anomaly recovery, all count positions from 200 to 206 will be marked as invalid, the atomic count cursor will be updated to 207, and the encrypted count intervals will be reallocated starting from 207.

[0178] This avoids the repeated use of the same encryption count position and candidate key stream under the same data group key before and after an anomaly occurs.

[0179] Encrypted subgroup index and ordered encapsulation

[0180] Create an encrypted subgroup index for each encrypted subgroup. The encrypted subgroup index should at least record the data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, encryption count position, authentication tag, and processing status.

[0181] The processing status can include pending encryption, encryption in progress, encryption completed, confirmed submission, and processing error.

[0182] Multiple encryption processing units can process different encryption subgroups in parallel, so a later encryption subgroup may be encrypted before a previous encryption subgroup.

[0183] Once the encryption subgroup has been completed, it can be written to the result cache and its processing status updated to encryption complete, without waiting for the preceding encryption subgroup to complete.

[0184] During the encapsulation process, encrypted compressed data belonging to the same data group are sorted according to the window identifier, encrypted subgroup sequence number, and compression start offset to restore the original order of each encrypted subgroup in the compressed byte stream of the data group.

[0185] For implementations that use the data group-level compression start offset, sorting can be done directly according to the data group-level compression start offset.

[0186] Group-level authentication information is generated based on the encrypted subgroup sequence number, compression start offset, effective data length, and authentication tag of each sorted encrypted subgroup.

[0187] Group-level authentication information can be generated using message authentication codes to detect instances where encrypted subgroups have been deleted, duplicated, replaced, or rearranged.

[0188] The final encrypted compression result can include task header information, data group header information, encrypted subgroup index, encrypted compressed data, authentication label, and group-level authentication information.

[0189] The task header information can record the processing task identifier, session random variable, compression algorithm identifier, encryption algorithm identifier, and parameter version. The data group header information can record the data group identifier, number of encrypted subgroups, original data group length, and compressed data group length.

[0190] Decryption and decompression process

[0191] After the reading end obtains the encrypted and compressed result, it generates the corresponding data group key using the same key derivation function as the encryption end, based on the processing task identifier, data group identifier, and session randomness.

[0192] Read the window identifier, encryption subgroup number, compression start offset, effective data length, encryption count position, and authentication tag of each encryption subgroup based on the encryption subgroup index.

[0193] The authentication and encryption modes corresponding to the encryption end are used to authenticate, verify, and decrypt each encrypted subgroup.

[0194] Multiple encrypted subgroups can be authenticated and decrypted in parallel. Encrypted subgroups that fail authentication are not included in subsequent concatenation and an integrity error message is output.

[0195] For encrypted subgroups that have passed authentication and verification, they are arranged in order according to the compression start offset and spliced ​​together to form a continuous compressed byte stream corresponding to the current data group.

[0196] For the end-encrypted subgroup, retain the valid compressed data according to its effective data length and remove the padding data.

[0197] Decompress the continuous compressed byte stream according to the compression algorithm identifier and compression parameter version to restore the original data of the current data group.

[0198] Finally, according to the data group identifier and arrangement order, the original data of each data group are spliced ​​together to restore the complete data to be processed.

[0199] System Implementation

[0200] This embodiment also provides a data compression and block encryption system that can be executed in parallel, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a data partitioning module, a compression output prediction module, a key stream parameter determination module, a parallel processing module, an encryption subgroup generation module, a block encryption module, a key stream adjustment module, and a result encapsulation module.

[0201] The data partitioning module is used to divide the data to be processed into multiple data groups, divide each data group into multiple compressed windows, and configure data group identifiers and window identifiers.

[0202] The compression output prediction module is used to extract data features of the current compression window, calculate the predicted compression ratio and predicted compressed output length, and determine the expected number of encrypted packets.

[0203] The key stream parameter determination module is used to generate a data group key based on the master key, processing task identifier, data group identifier and session random quantity, allocate encryption count intervals through atomic counting cursors, and determine candidate key stream generation parameters.

[0204] The parallel processing module is used to start the compression processing task and the candidate key stream generation task in parallel. It writes the variable-length compressed data generated by the compression processing task into the compression output buffer and writes the candidate key stream generated by the candidate key stream generation task into the key stream buffer.

[0205] The encrypted subgroup generation module determines the unencrypted compressed data based on the compressed write position and encrypted read position in the compressed output buffer. During the compression of the current data group, unencrypted compressed data that reaches the preset encrypted group length is formed into encrypted subgroups. After the current data group is compressed, the remaining unencrypted compressed data that is less than the preset encrypted group length is formed into a tail encrypted subgroup. The tail encrypted subgroup is a type of encrypted subgroup.

[0206] The block encryption module is used to initiate encryption processing before the current data group is fully compressed, and to perform block cipher-based authentication encryption processing on each encrypted subgroup that has been formed and subsequently formed.

[0207] The key stream adjustment module is used to allocate an extended count interval and generate an extended key stream when the actual compressed output length is greater than the predicted compressed output length, and to mark the encryption count position corresponding to the unused candidate key stream as invalid when the actual compressed output length is less than the predicted compressed output length.

[0208] The result encapsulation module is used to create an encrypted subgroup index, sort the encrypted compressed data that has been encrypted out of order, and encapsulate it in an ordered manner according to the original compressed byte stream order.

[0209] The system may also include a scheduling controller and a joint state management module. The scheduling controller generates encryption-priority processing signals, compression-priority processing signals, or compression pause signals based on the buffer occupancy ratio of the compressed output buffer. The joint state management module generates joint state checkpoints and restores the compression and encryption states in the event of an anomaly.

[0210] This embodiment reduces the processing delay caused by waiting for the entire data group to be compressed before starting encryption by overlapping the compression process, candidate key stream generation, and block encryption in the same data group. At the same time, it ensures a secure and continuous correspondence between variable-length compressed output and block encryption by using compression output prediction, dynamic adjustment of encryption count interval, buffer load adjustment, tail encryption subgroup processing, and joint state checkpointing.

[0211] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data compression and block encryption method that can be executed in parallel, characterized in that, Includes the following steps: S1. Obtain the data to be processed, divide the data to be processed into multiple sequentially arranged data groups, and divide each data group into multiple compressed windows, and configure data group identifiers and window identifiers for each data group and each compressed window. S2. Extract the data features of the current compression window, predict the compression output length of the current compression window based on the data features, and determine the expected number of encrypted packets and the encryption count interval corresponding to the expected number of encrypted packets based on the predicted compression output length. S3. Generate the data group key corresponding to the current data group based on the master key, processing task identifier, data group identifier of the current data group and session random number, and determine the candidate key stream generation parameters based on the data group key, the window identifier of the current compression window and the encryption count interval. S4. Start the compression processing task and the candidate key stream generation task of the current data group in parallel. The compression processing task reads the original data in the current data group in sequence, writes the generated variable-length compressed data into the compression output buffer continuously, and generates a candidate key stream according to the candidate key stream generation parameters through the candidate key stream generation task, and writes the candidate key stream into the key stream buffer. S5. Based on the compression write position and encryption read position in the compression output buffer, continuously determine the unencrypted compressed data corresponding to the current compression window; before all the original data of the current data group is compressed, when the unencrypted compressed data reaches the preset encryption group length, extract compressed data from it to form an encrypted subgroup; when all the original data of the current data group is compressed and there is remaining unencrypted compressed data that is less than the preset encryption group length, form the remaining unencrypted compressed data into a tail encrypted subgroup; configure the encrypted subgroup sequence number, compression start offset, and effective data length for the encrypted subgroup and the tail encrypted subgroup respectively; the tail encrypted subgroup belongs to the encrypted subgroup; S6. Determine the corresponding encryption count position according to the compression start offset of each encryption subgroup. Before all the original data of the current data group is compressed, start the encryption process for the formed encryption subgroups. Use the data group key and the candidate key stream of the corresponding encryption count position to perform block cipher-based encryption process on each formed and subsequently formed encryption subgroup to obtain encrypted compressed data and authentication tags. S7. Obtain the actual compressed output length of the current compression window, and adjust the candidate key stream according to the difference between the actual compressed output length and the predicted compressed output length. When the candidate key stream is insufficient, allocate an extended counting interval and generate an extended key stream. When there is a remaining candidate key stream, mark the encryption counting position corresponding to the unused candidate key stream as invalid. S8. Establish an encryption subgroup index based on the data group identifier, window identifier, encryption subgroup sequence number and compression start offset of each encryption subgroup, and encapsulate each encrypted compressed data in an orderly manner according to the encryption subgroup index to generate the encryption and compression result corresponding to the data to be processed.

2. The data compression and block encryption method that can be executed in parallel according to claim 1, characterized in that, The data features include at least the normalized data entropy of the current compression window, the compression dictionary matching rate, the proportion of literal data, and the actual compression rate of the previous compression window; the predicted compression rate of the current compression window is determined according to the following formula: in, For the first The predicted compression ratio for each compression window. This represents the actual compression ratio of the previous compression window. For the first Normalized data entropy of a compressed window, For the first The matching rate of the compressed dictionary for each compressed window. For the first The proportion of literal data in each compressed window. , , and The weights are preset and the sum of the weights is 1. and These are the preset lower compression ratio limit and the preset upper compression ratio limit, respectively. This means that the calculation results will be limited to between the preset lower limit and the preset upper limit of compression ratio; For the first compression window in the current data group, the actual compression ratio of the previous compression window is taken as the preset initial compression ratio.

3. The data compression and block encryption method that can be executed in parallel according to claim 2, characterized in that, The prediction error of the current compression window is determined based on the difference between the predicted compression ratio and the actual compression ratio of the compression windows preceding the current compression window; the predicted compression output length of the current compression window is determined according to the following formula: in, For the first The predicted compressed output length of each compression window. For the first The original data length of each compressed window. For the first The prediction error of each compressed window, This is the preset error amplification factor. To preset the encrypted packet length, This indicates rounding up; the expected number of encrypted packets is determined by rounding up the ratio of the predicted compressed output length to the preset encrypted packet length; for the first compression window in the current data group, the prediction error is taken as the preset initial prediction error.

4. The data compression and block encryption method that can be executed in parallel according to claim 1, characterized in that, A non-overlapping encryption count interval is allocated to each compression window using an atomic counting cursor. The atomic counting cursor value before allocation is used as the starting count value corresponding to the current compression window. After allocation, the atomic counting cursor is updated according to the expected number of encryption blocks. The expected number of encryption blocks in the current compression window and the starting encryption count position corresponding to the j-th encryption subgroup are determined according to the following formula: , in, For the first The expected number of encrypted blocks per compression window. For the first The predicted compressed output length of each compression window. To preset the encrypted packet length, For the first The first compression window The starting encryption count position corresponding to each encrypted subgroup For the first The starting count value corresponding to each compression window For the first The encrypted subgroup in the th _ The compression start offset in the compression output corresponding to each compression window. This indicates rounding down to the nearest integer.

5. The data compression and block encryption method that can be executed in parallel according to claim 1, characterized in that, The compressed output cache is a circular cache, which is equipped with a compressed write pointer, an encrypted read pointer, and a committed pointer. The cache occupancy ratio is determined based on the amount of data between the compressed write pointer and the encrypted read pointer. When the cache occupancy ratio is greater than the preset high occupancy threshold, the scheduling controller generates an encryption priority processing signal and sends the encryption priority processing signal to the task queue selection interface, so that the encryption processing unit can read the encryption subgroup to be encrypted first in the next scheduling cycle, or send a compression pause signal to the compression processing unit to pause the new compression window from entering the compression process. When the cache occupancy ratio is less than a preset low occupancy threshold, the scheduling controller generates a compression priority processing signal and sends the compression priority processing signal to the task queue selection interface, so that the compression processing unit can prioritize reading the compression window to be compressed in the next scheduling cycle, or release the compression pause signal.

6. The data compression and block encryption method that can be executed in parallel according to claim 4, characterized in that, Performing block cipher-based encryption on the encrypted subgroup includes: determining the starting encryption count position corresponding to the encrypted subgroup based on the compression start offset and the preset encryption block length; setting at least one of the data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, and compression parameter information as additional authentication data; and using an authentication encryption mode with a counter structure, performing encryption on the encrypted subgroup based on the data group key, the starting encryption count position, and the additional authentication data to generate the encrypted compressed data and authentication tag; wherein different encrypted subgroups within the same data group correspond to different encryption count positions.

7. The data compression and block encryption method that can be executed in parallel according to claim 4, characterized in that, It also includes generating joint state checkpoints at preset checkpoint intervals. Each joint state checkpoint records at least the data group identifier, the position of the original data read, the window identifier of the current compression window, compression status information, compression write position, encryption read position, the sequence number of the submitted encryption subgroup, the atomic count cursor, and the length of the completed encryption data. When an anomaly occurs in the compression or encryption task, the most recently verified joint state checkpoint is restored, and the compressed data, encrypted compressed data, and corresponding candidate key streams that have not yet been confirmed and submitted after the joint state checkpoint are deleted. The allocated encryption count intervals after the joint state checkpoint are marked as invalid. The maximum end count position in the allocated encryption count intervals is determined, the atomic count cursor is updated to the count position after the maximum end count position, and the encryption count intervals and candidate key streams are reassigned from the updated atomic count cursor to continue executing the compression and encryption tasks.

8. The data compression and block encryption method that can be executed in parallel according to claim 1, characterized in that, The encrypted subgroup index records at least the data group identifier, window identifier, encrypted subgroup sequence number, compression start offset, effective data length, encryption count position, authentication tag, and processing status of each encrypted subgroup. Multiple encrypted subgroups belonging to the same data group are allowed to complete encryption processing in an order different from the encrypted subgroup sequence number. The encrypted compressed data that has undergone encryption processing is sorted and encapsulated according to the window identifier, encrypted subgroup sequence number, and compression start offset. Group-level authentication information is generated based on the encrypted subgroup sequence number, compression start offset, effective data length, and authentication tag of each sorted encrypted subgroup.

9. A data compression and block encryption system capable of parallel execution, characterized in that, It includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the following functional modules: The data partitioning module is used to acquire the data to be processed, divide the data to be processed into multiple sequentially arranged data groups, and divide each data group into multiple compressed windows; The compression output prediction module is used to predict the compression output length based on the data characteristics of the current compression window, and to determine the expected number of encrypted packets and the encryption count interval based on the predicted compression output length. The key stream parameter determination module is used to generate the data group key corresponding to the current data group, and determine the candidate key stream generation parameters based on the data group key, window identifier and encryption count interval; The parallel processing module is used to start compression processing tasks and candidate key stream generation tasks in parallel, write the generated variable-length compressed data into the compression output buffer, and write the generated candidate key stream into the key stream buffer. The encrypted subgroup generation module is used to form encrypted subgroups based on the unencrypted compressed data in the compressed output buffer before all the original data of the current data group is compressed. The block encryption module is used to determine the corresponding encryption count position based on the compression start offset of the encrypted subgroup before all the original data of the current data group is compressed, and to perform block cipher-based encryption processing on the encrypted subgroup using the data group key and the candidate key stream corresponding to the encryption count position. The keystream adjustment module is used to generate an extended keystream based on the difference between the actual compressed output length and the predicted compressed output length, or to mark the encryption count position corresponding to the unused candidate keystream as invalid. The result encapsulation module is used to establish an encrypted subgroup index based on the data group identifier, window identifier, encrypted subgroup sequence number and compression start offset of each encrypted subgroup, and to encapsulate each encrypted compressed data in an orderly manner according to the encrypted subgroup index.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 8.

Images