An end-to-end encrypted streaming method with multi-threaded resume support and local reassembly

Abstract

The application provides an end-to-end encryption streaming and local reorganization method supporting multi-threaded breakpoint continuation, and specifically comprises the following steps: a server performs streaming encryption and unpacking-free uploading of a data set into a lake; the server obtains a non-symmetric encryption public key provided by a client, generates and issues a multi-file logical list; a client reads and analyzes the list, divides multiple data blocks, and issues a range request; the client dynamically calculates an AES-CTR counter according to a ciphertext block start offset for each ciphertext block, independently generates a key stream for exclusive OR decryption, and simultaneously randomly writes plaintext into a corresponding offset in the local and records the written position. The technical scheme of the application overcomes the problems of waste of server computing power, conflict between block encryption algorithm and concurrent transmission in the prior art.

Description

Technical Field

[0001] This invention relates to the field of encrypted data transmission, and specifically to an end-to-end encrypted streaming transmission and local reassembly method that supports multi-threaded breakpoint resumption. Background Technology

[0002] In the fields of commercial aerospace and massive telemetry, tracking, and command (TT&C) data processing, the datasets produced by data production systems for a single mission typically contain multiple files. For example, a single SAR image usually includes a TIFF image file of several gigabytes in size and an XML metadata file. Existing data distribution and encrypted download technologies have the following significant drawbacks: 1. Server-side computing power waste and pipeline blockage: Traditional solutions typically physically package multiple files into ZIP / TAR packages on the server before providing download functionality. In a business pipeline involving satellite overpasses, real-time imaging, and automatic uploads, forced packaging and compression consume a significant amount of CPU and memory on computing nodes. More importantly, encrypted ciphertext data has extremely high information entropy, making traditional compression algorithms largely ineffective and potentially leading to file size inflation.

[0003] 2. Conflict between block encryption algorithms and concurrent transmission: Existing object storage end-to-end encryption typically uses standard block encryption modes (such as AES-CBC). Block encryption has a sequential dependency during decryption, requiring preceding data blocks to proceed. This prevents clients from accessing random locations by initiating HTTP Range requests when downloading large files, thus completely losing the ability for multi-threaded concurrent downloading and resuming interrupted downloads, severely reducing the efficiency of distributing massive amounts of data.

[0004] Therefore, there is a need for an end-to-end encrypted streaming method that can shorten the latency of data entry and distribution, enable random access to end-to-end encrypted files, support multi-threaded breakpoint resumption, and local reassembly. Summary of the Invention

[0005] The main objective of this invention is to provide an end-to-end encrypted streaming transmission and local reassembly method that supports multi-threaded breakpoint resumption, in order to solve the problems of server-side computing power waste and block encryption algorithm conflict with concurrent transmission in the prior art.

[0006] To achieve the above objectives, this invention provides an end-to-end encrypted streaming transmission and local reassembly method that supports multi-threaded breakpoint resumption, specifically including the following steps: S1, the server performs streaming encryption on the dataset and uploads it to the lake without packaging.

[0007] S2: The server obtains the asymmetric encryption public key provided by the client, generates and distributes a multi-file logical list.

[0008] S3: The client reads and parses the manifest, divides it into multiple data blocks, and issues a range request.

[0009] S4: For each ciphertext block, the client dynamically calculates the AES-CTR counter based on the starting offset of the ciphertext block and independently generates a key stream for XOR decryption. At the same time, the plaintext is randomly written to the corresponding offset locally and the written position is recorded.

[0010] Furthermore, step S1 specifically includes the following steps: S1.1, the remote sensing satellite ground station system completes real-time imaging and produces a standard dataset including TIFF files and auxiliary XML files.

[0011] S1.2, the data access service of the measurement and control center generates random AES symmetric keys and random numbers for TIFF files and XML files respectively without performing ZIP / TAR physical merging.

[0012] S1.3 uses a counter mode to perform streaming encryption on TIFF and XML files; the encryption process and the upload of the Minimum Object Storing (MinIO) are performed concurrently using Pipeline streaming processing.

[0013] Furthermore, step S2 specifically includes the following steps: S2.1 The server obtains the asymmetric encryption public key provided by the client and uses the asymmetric encryption public key to perform asymmetric encryption on the AES symmetric key of the TIFF file and the XML file respectively.

[0014] S2.2, the server calls the MinIO application programming interface (API) to generate a pre-signed URL with an expiration time for each encrypted object.

[0015] S2.3 The server encapsulates the file name, pre-signed URL, random number, and AES symmetric key encrypted with the public key into a JSON-formatted manifest and sends it to the client.

[0016] Furthermore, step S3 specifically includes the following steps: S3.1, the client software reads and parses the manifest, and creates a folder with the same name on the local disk corresponding to the real-time images acquired by the remote sensing satellite ground station system.

[0017] S3.2 The client uses its locally secure and isolated private key to decrypt the real AES symmetric key of the TIFF and XML files.

[0018] S3.3 For TIFF files, the client logically divides the TIFF file into multiple blocks of fixed size based on the current network environment.

[0019] S3.4 The client starts multiple threads to send range requests to the pre-signed URL for different data blocks and concurrently pulls ciphertext blocks.

[0020] Furthermore, step S4 specifically includes the following steps: S4.1 When any ciphertext block arrives at the client's memory, the client dynamically calculates the counter value corresponding to the starting byte offset in the AES-CTR encryption mode based on the starting byte offset of the ciphertext block.

[0021] S4.2, the client uses the real AES symmetric key and the calculated current counter to independently generate the key stream required for the ciphertext block in step S4.1, and directly XORs the ciphertext block to decrypt it into plaintext.

[0022] S4.3 The client uses the operating system's random file write API to write the decrypted plaintext blocks directly to the corresponding byte offsets in the local TIFF file.

[0023] S4.4 When a network interruption occurs, the client only needs to record the position of the bytes that have been successfully written locally; after reconnecting to the network, it can directly continue to initiate a range request from the offset of the interruption and calculate the corresponding key stream to complete the breakpoint resume transmission in end-to-end encrypted state.

[0024] The present invention has the following beneficial effects: 1. This invention greatly improves the throughput of the server pipeline: It completely eliminates the invalid ZIP compression step for high-entropy encrypted data on the server side, reduces the CPU / memory overhead of the server, and makes it possible to upload encrypted streaming immediately after imaging is completed, which greatly shortens the latency of data entry and distribution.

[0025] 2. This invention perfectly supports multi-threaded concurrency and breakpoint resumption: it overcomes the technical bias that traditional end-to-end encrypted files cannot be randomly accessed. Even if the file is encrypted throughout the process, the client can still make full use of the operating system's underlying network I / O performance, significantly improving download speed through multi-threaded block fetching, and achieving seamless resumption of downloads after network interruption.

[0026] 3. Seamless restructuring of the client experience: The client automatically creates a directory locally by parsing the Manifest and concurrently pulls all relevant files (such as TIFF and XML), providing customers with an ultimate one-click full download experience without the need for physical packaging on the server side. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of an end-to-end encrypted streaming transmission and local reassembly method supporting multi-threaded breakpoint resumption is shown. Detailed Implementation

[0028] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1 The method for end-to-end encrypted streaming and local reassembly that supports multi-threaded breakpoint resumption includes the following steps: S1, the server performs streaming encryption on the dataset and uploads it to the lake without packaging.

[0030] S2: The server obtains the asymmetric encryption public key provided by the client, generates and distributes a multi-file logical list.

[0031] S3: The client reads and parses the manifest, divides it into multiple data blocks, and issues a range request.

[0032] S4: For each ciphertext block, the client dynamically calculates the AES-CTR counter based on the starting offset of the ciphertext block and independently generates a key stream for XOR decryption. At the same time, the plaintext is randomly written to the corresponding local offset and the written position is recorded. This allows the client to continue decryption and writing by recalculating the counter from the breakpoint offset after a network interruption, thereby achieving secure breakpoint resume transmission in end-to-end encrypted state.

[0033] Specifically, step S1 includes the following steps: S1.1, the remote sensing satellite ground station system completes real-time imaging and produces a standard dataset including a large-volume TIFF file and auxiliary XML files; S1.2, The data access service of the measurement and control center generates random AES symmetric keys and random numbers IV / Nonce for TIFF files and XML files respectively without performing ZIP / TAR physical merging; S1.3 uses AES-CTR in counter mode to perform streaming encryption on TIFF and XML files; the encryption process and the upload of the Minimum Object Memory (MinIO) are performed concurrently using Pipeline streaming processing.

[0034] Specifically, step S2 includes the following steps: S2.1, The server obtains the asymmetric encryption public key provided by the client and uses the asymmetric encryption public key to perform asymmetric encryption on the AES symmetric key of the TIFF file and the XML file respectively; S2.2, the server calls the MinIO application programming interface API to generate a pre-signed URL with an expiration time for each encrypted object; S2.3 The server encapsulates the file name, pre-signed URL, random number, and AES symmetric key encrypted with the public key into a JSON-formatted manifest (i.e., logical packaging) and sends it to the client.

[0035] Specifically, step S3 includes the following steps: S3.1, the client software (such as a standalone downloader developed based on Python / Tauri) reads and parses the manifest, and creates a folder with the same name on the local disk corresponding to the real-time images obtained by the remote sensing satellite ground station system; S3.2, the client uses its locally secure and isolated private key to decrypt the real AES symmetric key of the TIFF and XML files; S3.3 For TIFF files, the client logically divides the TIFF file into multiple blocks of fixed size (e.g., each block is 16MB) based on the current network environment. S3.4 The client starts multiple threads to send range requests to the pre-signed URL for different data blocks, i.e., HTTP requests with Range: bytes=start-end, and concurrently pulls ciphertext blocks.

[0036] Specifically, step S4 includes the following steps: S4.1 When any ciphertext block arrives at the client's memory, the client dynamically calculates the counter value corresponding to the starting byte offset in the AES-CTR encryption mode based on the starting byte offset of the ciphertext block. S4.2, the client uses the real AES symmetric key and the calculated current counter to independently generate the key stream required for the ciphertext block in step S4.1, and directly XORs the ciphertext block to decrypt it into plaintext; S4.3 The client uses the operating system's random file write API to write the decrypted plaintext blocks directly to the corresponding byte offsets of the local TIFF file; S4.4 When a network interruption occurs, the client only needs to record the position of the bytes that have been successfully written locally; after reconnecting to the network, it can directly continue to initiate a range request from the offset of the interruption and calculate the corresponding key stream to complete the breakpoint resume transmission in end-to-end encrypted state.

[0037] This invention proposes an end-to-end encrypted streaming method that supports multi-threaded breakpoint resumption and local reassembly. It has the following beneficial effects: 1. Eliminate physical packaging and introduce a logical manifest: After data is generated (such as TIFF and XML), it is not physically merged. Instead, the server generates independent encrypted files and uploads them to object storage (such as MinIO). The server generates a logical manifest containing the pre-signed URLs of each file and their corresponding encryption keys and delivers it to the client.

[0038] 2. The underlying layer adopts a streaming encryption mode (such as AES-CTR): Before the data is uploaded to the lake, the server uses streaming encryption algorithms such as AES-CTR (counter mode) to encrypt the file.

[0039] 3. The client dynamically calculates the keystream based on the offset: After parsing the logical manifest, the client initiates an HTTP Range request for the ciphertext to the object storage according to the locally set multi-threaded block size. Using the extracted initial counter (Nonce / IV) and the offset of the currently requested byte, the client independently calculates the AES-CTR keystream corresponding to the data block, directly decrypts the out-of-order ciphertext block in memory, and writes it to the local disk according to the offset.

[0040] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for end-to-end encrypted streaming transmission and local reassembly that supports multi-threaded breakpoint resumption, characterized in that, Specifically, the steps include the following: S1, the server performs streaming encryption on the dataset and uploads it to the lake without packaging; S2: The server obtains the asymmetric encryption public key provided by the client, generates and distributes a multi-file logical list; S3: The client reads and parses the manifest, divides it into multiple data blocks, and issues a range request; S4: For each ciphertext block, the client dynamically calculates the AES-CTR counter based on the starting offset of the ciphertext block and independently generates a key stream for XOR decryption. At the same time, the plaintext is randomly written to the corresponding offset locally and the written position is recorded.

2. The end-to-end encrypted streaming transmission and local reassembly method supporting multi-threaded breakpoint resumption as described in claim 1, characterized in that, Step S1 specifically includes the following steps: S1.1, the remote sensing satellite ground station system completes real-time imaging and produces a standard dataset including TIFF files and auxiliary XML files; S1.2, The data access service of the measurement and control center generates random AES symmetric keys and random numbers for TIFF files and XML files respectively without performing ZIP / TAR physical merging; S1.3 uses a counter mode to perform streaming encryption on TIFF and XML files; the encryption process and the upload of the Minimum Object Memory (MinIO) are performed concurrently using Pipeline streaming processing.

3. The end-to-end encrypted streaming transmission and local reassembly method supporting multi-threaded breakpoint resumption as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S2.1, The server obtains the asymmetric encryption public key provided by the client and uses the asymmetric encryption public key to perform asymmetric encryption on the AES symmetric key of the TIFF file and the XML file respectively; S2.2, the server calls the MinIO application programming interface API to generate a pre-signed URL with an expiration time for each encrypted object; S2.3 The server encapsulates the file name, pre-signed URL, random number, and AES symmetric key encrypted with the public key into a JSON-formatted manifest and sends it to the client.

4. The end-to-end encrypted streaming transmission and local reassembly method supporting multi-threaded breakpoint resumption as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S3.1, the client software reads and parses the manifest, and creates a folder with the same name on the local disk corresponding to the real-time images obtained by the remote sensing satellite ground station system; S3.2, the client uses its locally secure and isolated private key to decrypt the real AES symmetric key of the TIFF and XML files; S3.3 For TIFF files, the client logically divides the TIFF file into multiple blocks of fixed size according to the current network environment; S3.4 The client starts multiple threads to send range requests to the pre-signed URL for different data blocks and concurrently pulls ciphertext blocks.

5. The end-to-end encrypted streaming transmission and local reassembly method supporting multi-threaded breakpoint resumption as described in claim 1, characterized in that, Step S4 specifically includes the following steps: S4.1 When any ciphertext block arrives at the client's memory, the client dynamically calculates the counter value corresponding to the starting byte offset in the AES-CTR encryption mode based on the starting byte offset of the ciphertext block. S4.2, the client uses the real AES symmetric key and the calculated current counter to independently generate the key stream required for the ciphertext block in step S4.1, and directly XOR the ciphertext block to decrypt it into plaintext; S4.3 The client uses the operating system's random file write API to write the decrypted plaintext blocks directly to the corresponding byte offsets of the local TIFF file; S4.4 When a network interruption occurs, the client only needs to record the position of the bytes that have been successfully written locally; after reconnecting to the network, it can directly continue to initiate a range request from the offset of the interruption and calculate the corresponding key stream to complete the breakpoint resume transmission in end-to-end encrypted state.

Images