A method, apparatus, device, and storage medium for transmitting large files.

Abstract

This application discloses a method, apparatus, device, and storage medium for large file transmission, relating to the field of computer storage and network transmission technology. The method includes: allocating independent virtual threads to each segment of file content; using the virtual threads to perform parallel compression on each segment of file content to obtain segmented compressed data; employing a two-layer key derivation algorithm to determine a segment-level derivation key and a one-time random number for each segment of file content; encrypting the segmented compressed data using the virtual threads to obtain encrypted segmented data for each segment of file content and uploading it to an object storage space; and, upon successful uploading of all segmented file content, obtaining signed encrypted metadata and uploading it to the object storage space, and binding it to the encrypted segmented data. This scheme, with its two-layer key structure, intelligent dynamic compression, and parallel streaming processing via virtual threads, improves the efficiency and security of large file transmission.

Description

Technical Field

[0001] This application relates to the field of computer storage and network transmission technology, specifically to a method, apparatus, device, and storage medium for transmitting large files. Background Technology

[0002] With the deepening of digital transformation, the demand for storing and downloading large files such as high-definition videos, backup files, and engineering data has exploded.

[0003] Existing file processing solutions mostly employ overall file encryption or fixed-key fragment encryption, which poses a risk of key leakage and requires loading the entire file or a large number of fragments into memory. This results in a trade-off between processing efficiency and resource consumption, and significant concurrent performance bottlenecks. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for transmitting large files, so as to improve the efficiency and security of large file transmission.

[0005] According to one aspect of this application, a method for transferring large files is provided, the method comprising: Based on the file attribute information of the file to be stored and the preset memory threshold, the file content of the file to be stored is split into pieces to obtain at least two pieces of file content and corresponding piece indexes. An independent virtual thread is allocated to the content of each file segment. An adaptive dynamic compression algorithm is adopted. Based on the file attribute information of each file segment, the content of each file segment is compressed in parallel through the virtual thread to obtain the compressed data of each file segment. A two-layer key derivation algorithm is adopted to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user's input password and the fragment index of each fragment file content. Using the fragment-level derived key and the one-time random number, the fragment compressed data is encrypted through the virtual thread to obtain the fragment encrypted data of each fragment file content, and the fragment encrypted data is uploaded to the object storage space; Once all file fragments have been successfully uploaded, the file-level metadata and fragment-level metadata of the file content are digitally signed and encrypted to obtain signed encrypted metadata. This signed encrypted metadata is then uploaded to the object storage space and bound to the fragment encrypted data.

[0006] According to another aspect of this application, a large file transfer device is provided, the device comprising: The sharding module is used to shard the file content of the file to be stored according to the file attribute information and the preset memory threshold, so as to obtain at least two shard file contents and the corresponding shard index. The segmented compression module is used to allocate independent virtual threads for each segment file content. It adopts an adaptive dynamic compression algorithm and performs parallel compression on each segment file content through the virtual threads according to the file attribute information of each segment file content to obtain segmented compressed data of each segment file content. The data determination module is used to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user's input password and the fragment index of each fragment file content using a two-layer key derivation algorithm. The fragment encryption module is used to encrypt the fragment compressed data through the virtual thread using the fragment-level derived key and the one-time random number to obtain the fragment encrypted data of each fragment file content, and upload the fragment encrypted data to the object storage space; The data upload module is used to digitally sign and encrypt the file-level metadata and fragment-level metadata of the file content after all fragment file contents have been successfully uploaded, to obtain signed encrypted metadata, and upload the signed encrypted metadata to the object storage space, and bind it to the fragment encrypted data.

[0007] According to another aspect of this application, an electronic device is provided, the electronic device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the large file transfer methods provided in the embodiments of this application.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the large file transfer methods provided in the embodiments of this application.

[0009] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the large file transfer methods provided in the embodiments of this application.

[0010] This application segments the file content of the file to be stored based on its file attribute information and a preset memory threshold, obtaining at least two segments and their corresponding indexes. An independent virtual thread is allocated to each segment, and an adaptive dynamic compression algorithm is used to compress each segment in parallel using the virtual thread, based on its file attribute information, resulting in compressed segment data. A two-layer key derivation algorithm is employed, using the user-input password and the segment index to determine the segment-level derivation key and a one-time random number for each segment. The compressed segment data is then encrypted using the segment-level derivation key and the one-time random number via the virtual thread, resulting in encrypted segment data, which is uploaded to the object storage space. Once all segment data has been successfully uploaded, the file-level metadata and segment-level metadata of the file content are digitally signed and encrypted to obtain signed encrypted metadata, which is then uploaded to the object storage space and bound to the encrypted segment data. The above scheme dynamically adjusts the fragment size based on file attribute information and preset hardware memory thresholds; it employs an adaptive dynamic compression algorithm for parallel intelligent compression, balancing compression ratio and processing speed; it designs a two-layer key structure to improve the security of large file transfers; and it uses a customized virtual thread pool to allocate an independent virtual thread to each fragment, achieving deep binding between the virtual thread and the fragment lifecycle, reducing context switching overhead, and improving the efficiency of large file transfers. Attached Figure Description

[0011] Figure 1 This is a flowchart of a method for transferring large files according to Embodiment 1 of this application; Figure 2 This is a flowchart of a method for transferring large files according to Embodiment 2 of this application; Figure 3 This is a schematic diagram of a large file transmission device according to Embodiment 3 of this application; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the large file transmission method of Embodiment 4 of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0013] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0014] Furthermore, it should be noted that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0015] Example 1 Figure 1 This is a flowchart of a large file transmission method according to Embodiment 1 of this application. This embodiment is applicable to the intelligent compression and encryption transmission of large files based on virtual threads, particularly suitable for high-concurrency, low-memory, high-security storage and cross-network transmission scenarios of hundreds of gigabytes of ultra-large files. It can be widely used in business scenarios such as large file distribution on video platforms, enterprise-level data backup and synchronization, cloud storage security enhancement, cross-regional large file sharing, and big data and artificial intelligence training data processing. It can be executed by a large file transmission device, which can be implemented in hardware and / or software and can be configured in computer equipment, such as a server. Figure 1 As shown, the method includes: S110. Based on the file attribute information of the file to be stored and the preset memory threshold, the file content of the file to be stored is segmented to obtain at least two segmented file contents and corresponding segment indexes.

[0016] The file to be stored refers to the original file that the user wants to upload and save, especially large files in the hundreds of gigabytes range. File attribute information for the file to be stored may include its type, redundancy, and size. The file type may include videos, documents, images, compressed files, etc. Redundancy is the ratio of the amount of duplicate data eliminated by data compression algorithms in a unit data block to the total amount of original data, which can be calculated using information entropy or determined based on dictionary-based repeating pattern matching. The preset memory threshold is a user-defined upper limit for memory usage to prevent server memory exhaustion and system crashes during large file processing, ensuring service stability.

[0017] Specifically, the system verifies the authentication information carried in file upload requests. Upon successful verification, the file to be stored is received, ensuring that only authorized users can upload files and preventing unauthorized access and data corruption. Then, the system analyzes file type and data redundancy, and, in conjunction with configured memory thresholds, segments the file content to be stored into chunks. The chunk size is dynamically adjusted (default 20MB, adjustable from 10-50MB depending on file characteristics), and the number of chunks is calculated to avoid compatibility issues with fixed chunks.

[0018] S120. Allocate an independent virtual thread for each segment file content, adopt an adaptive dynamic compression algorithm, and perform parallel compression of each segment file content through the virtual thread according to the file attribute information of each segment file content to obtain segment compressed data of each segment file content.

[0019] In large-scale concurrent file processing, traditional thread pool management methods are difficult to meet the performance and stability requirements of processing millions of files due to the high cost of thread resources and limited concurrency. Therefore, this application embodiment utilizes the virtual thread mechanism introduced in Java 21 to allocate an independent virtual thread for each piece of file content without increasing the thread management burden.

[0020] Specifically, a virtual thread pool is initialized, the concurrency level is set according to hardware resources, and an independent virtual thread is allocated to each shard, thus binding the virtual thread to the lifecycle of each shard file content. The content of a single shard file is processed in an independent virtual thread without disk caching, and only one shard is loaded into memory to ensure constant memory usage. The data characteristics of each shard file content are analyzed, and an adaptive dynamic compression algorithm is used to compress each shard file content in parallel to obtain the sharded compressed data of each shard file content.

[0021] Optionally, an adaptive dynamic compression algorithm is employed. Based on the file attribute information of each file segment, the virtual thread performs parallel compression on the contents of each file segment to obtain the compressed data of each file segment. This includes: determining the current compression level of each file segment based on its type and redundancy; and employing the adaptive dynamic compression algorithm to perform parallel compression on the contents of each file segment using the virtual thread based on the current compression level to obtain the compressed data of each file segment.

[0022] The adaptive dynamic compression algorithm automatically adjusts the Zstd (Zstandard) compression level based on the characteristics of the fragmented data, achieving an optimal balance between compression ratio and processing speed. Specifically, it performs a two-dimensional judgment based on the type and redundancy of the fragmented file content to avoid misjudgment based on a single dimension; the compression level can be divided into three levels, taking into account both compression ratio and processing speed; the compression level is stored in fragment-level metadata, and adaptation parameters are used during decompression to ensure the accuracy of decompression.

[0023] Optionally, determining the current compression level of each segment file content based on its type and redundancy includes: if the current segment file content is of a first type and its redundancy is less than a first redundancy threshold, then the current compression level of the current segment file content is determined to be a first compression level; the first type is a video file or a compressed file; if the current segment file content is of a second type, its redundancy is greater than or equal to the first redundancy threshold and less than a second redundancy threshold, then the current compression level of the current segment file content is determined to be a second compression level; the second type is different from both the first and third types; if the current segment file content is of a third type and its redundancy is greater than or equal to the second redundancy threshold, then the current compression level of the current segment file content is determined to be a third compression level; the third type is a text file or a document file; wherein the first redundancy threshold is less than the second redundancy threshold.

[0024] The first type can be video or compressed file types, such as MP4, AVI, MKV, ZIP, RAR, 7Z, GZ, etc. The third type can be text or document types, such as TXT, DOC, PDF, HTML, XML, JSON, log files, etc. The second type differs from both the first and third types and can be any other type besides the first and third types, such as image types (JPG, PNG, GIF, BMP, etc.); audio types (MP3, WAV, FLAC, AAC, etc.); executable files (EXE, DLL, SO, JAR, etc.); and database types (DB, MDB, etc.). The first and second redundancy thresholds can be preset based on actual conditions or experience; this application does not specifically limit this. For example, the first redundancy threshold can be 30%, and the second redundancy threshold can be 60%.

[0025] S130. A two-layer key derivation algorithm is adopted to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user password input by the user and the fragment index of each fragment file content.

[0026] The dual-layer key derivation algorithm employs a two-layer system of file-level master keys and fragment-level derived keys to achieve independent encryption at the fragment level, preventing the entire file from being cracked due to a single key leak. Specifically, a file-level master key is generated based on PBKDF2WithHmacSHA256 (a combination algorithm of cryptographic key derivation function 2 and hash-based message authentication code-secure hash algorithm 256), ensuring the security of the master key. HKDF-SHA256 (an HMAC-based key extraction and expansion derivation function) is used to derive independent data keys from the master key for each fragment file content. Derivation parameters can include fragment index, fragment simple checksum, and random factor, ensuring that the key for each fragment file content is unique and cannot be guessed. Furthermore, the file-level master key is stored in ciphertext, and the fragment derivation parameters are stored in fragment-level metadata, which can only be recovered using the user's password, enhancing the security of the key system.

[0027] Optionally, the step of employing a two-layer key derivation algorithm to determine the fragment-level derivation key and one-time random number for each fragment file content based on the user-input password and the fragment index of each fragment file content includes: employing a two-layer key derivation algorithm to determine the file-level master key based on the user-input password and a randomly generated salt value, and employing a cryptographically secure pseudo-random number generator to generate a global random number for the file; and determining the fragment-level derivation key and one-time random number for each fragment file content based on the file-level master key, the global random number for the file, and the fragment index of each fragment file content.

[0028] The one-time random number (unique nonce) is constructed by combining an 8-byte global random number for the file, a 4-byte fragment index, and a 4-byte fragment file content checksum, forming a unique identifier for each fragment. This ensures that the encrypted random numbers for each fragment are not repeated, achieving fragment-level data integrity verification. The fragment file content checksum is a check value obtained by performing a lightweight digest operation on the current fragment file content. It is used to combine with the global random number for the file and the fragment index to construct a one-time random number, achieving a cryptographic binding between fragment data characteristics and the random number.

[0029] Specifically, based on the user's input password and a randomly generated salt value, the file-level master key is determined using PBKDF2WithHmacSHA256, and a cryptographically secure pseudo-random number generator is used to generate a global random number for the file. Then, based on the file-level master key, the global random number for the file, the fragment index of each fragment's content, and the checksum of each fragment's content, the fragment-level derived key and a one-time random number for each fragment's content are determined using HKDF-SHA256.

[0030] S140. Using the fragment-level derived key and the one-time random number, the fragment compressed data is encrypted through the virtual thread to obtain the fragment encrypted data of each fragment file content, and the fragment encrypted data is uploaded to the object storage space.

[0031] Specifically, an authentication encryption algorithm (AES-256-GCM) can be used. Based on the fragment-level derived key and a one-time random number, the fragmented compressed data is encrypted through a virtual thread to obtain the fragmented encrypted data of each fragment file and a 16-byte authentication tag. The encrypted fragmented data and authentication tag are then uploaded to the object storage space, and the fragment upload status (success, failure, or pending) is recorded. Incremental upload is supported.

[0032] S150. If all file fragment contents are successfully uploaded, digitally sign and encrypt the file-level metadata and fragment-level metadata of the file contents to obtain signed encrypted metadata, and upload the signed encrypted metadata to the object storage space and bind it to the fragment encrypted data.

[0033] File-level metadata may include a unique file identifier, original filename, original file size, total number of fragments, fragment size range, master key ciphertext, 16-byte salt value, 8-byte global random number, creation time, and tamper-proof signature. Fragment-level metadata may include fragment index, compressed size, compression level parameters, fragment-level derived key, one-time random number, authentication label, and fragment hash value.

[0034] Specifically, after all fragments are successfully processed, file-level metadata and fragment-level metadata of the file content can be generated. Then, the dual-layer metadata is signed with SHA256 (Secure Hash Algorithm 256) and encrypted before storage to ensure that the metadata is not tampered with. The encrypted and signed dual-layer metadata is uploaded to the object storage space, bound to the fragment file content, and the upload success result and the unique file identifier are returned to the client for subsequent download.

[0035] In one optional implementation, if the processing or upload of the current fragment file content fails, an exponential backoff retry mechanism is initiated. If the number of retries meets the exception handling conditions, the circuit breaker of the current virtual thread is triggered, and the failed fragment information of the current fragment file content is recorded.

[0036] The exception handling condition can be that the number of retries exceeds a preset threshold and the processing still fails or the upload fails. The preset threshold can be set manually based on actual conditions or experience. This application embodiment does not specifically limit this; for example, the preset threshold is 3 times.

[0037] Specifically, the status of shard processing or upload in each virtual thread can be monitored; when a shard processing or upload failure is detected, an exponential backoff retry mechanism is initiated for the failed shard; it is determined whether the number of retry attempts meets the exception handling conditions; if not, the shard processing and upload are re-executed after a delay of the exponential backoff interval; if so, the current virtual thread is triggered to terminate the shard task and release associated resources, and the information of the failed shard is recorded in the failed shard log, while the remaining shards continue to execute independently.

[0038] This application segments the file content of the file to be stored based on its file attribute information and a preset memory threshold, obtaining at least two segments and their corresponding indexes. An independent virtual thread is allocated to each segment, and an adaptive dynamic compression algorithm is used to compress each segment in parallel using the virtual thread, based on its file attribute information, resulting in compressed segment data. A two-layer key derivation algorithm is employed, using the user-input password and the segment index to determine the segment-level derivation key and a one-time random number for each segment. The compressed segment data is then encrypted using the segment-level derivation key and the one-time random number via the virtual thread, resulting in encrypted segment data, which is uploaded to the object storage space. Once all segment data has been successfully uploaded, the file-level metadata and segment-level metadata of the file content are digitally signed and encrypted to obtain signed encrypted metadata, which is then uploaded to the object storage space and bound to the encrypted segment data. The above scheme dynamically adjusts the fragment size based on file attribute information and preset hardware memory thresholds; it employs an adaptive dynamic compression algorithm for parallel intelligent compression, balancing compression ratio and processing speed; it designs a two-layer key structure to improve the security of large file transfers; and it uses a customized virtual thread pool to allocate an independent virtual thread to each fragment, achieving deep binding between the virtual thread and the fragment lifecycle, reducing context switching overhead, and improving the efficiency of large file transfers.

[0039] Example 2 Figure 2This is a flowchart of a method for transferring large files according to Embodiment 2 of this application. Based on the technical solutions of the above embodiments, this embodiment adds the following: "According to the unique file identifier carried in the file download request, signature-encrypted metadata is obtained from the object storage space, and based on the signature-encrypted metadata, file-level metadata and fragment-level metadata of the file content are determined; wherein, the signature-encrypted metadata is obtained by digitally signing and encrypting the file-level metadata and fragment-level metadata of the file content; a two-layer key derivation algorithm is adopted to determine the fragment-level derivation key and a one-time random number for each fragment of the file content based on the user-input password, the file-level metadata, and the fragment-level metadata; an independent virtual thread is allocated to each fragment of the file content, and the fragment-level encryption key for each fragment of the file content is obtained from the object storage space through the virtual thread." The data is encrypted, and multi-level integrity checks are performed on the encrypted data and the fragment-level metadata to obtain multi-level integrity check results. The encrypted data is obtained by encrypting the compressed data, which is obtained by parallel compression of the contents of each fragment file using an adaptive dynamic compression algorithm. If the multi-level integrity check results pass, the encrypted data is decrypted using a virtual thread based on the fragment-level derived key and the one-time random number to obtain the compressed data of each fragment file content. An adaptive dynamic decompression algorithm is then used to decompress the compressed data based on the compression level parameters in the fragment-level metadata to obtain the contents of each target fragment file. The target file is determined based on the contents of each target fragment file and the corresponding fragment index. It should be noted that parts not detailed in this embodiment can be found in other embodiments. Figure 2 As shown, the method includes: S210. Based on the unique file identifier carried in the file download request, obtain the signed and encrypted metadata from the object storage space, and determine the file-level metadata and fragment-level metadata of the file content based on the signed and encrypted metadata; wherein, the signed and encrypted metadata is obtained by digitally signing and encrypting the file-level metadata and fragment-level metadata of the file content.

[0040] Specifically, the authentication information carried in the file download request is verified. After successful verification, the signed and encrypted metadata is downloaded from the object storage space based on the unique file identifier carried in the file download request. First, SHA256 signature verification is performed to ensure that the metadata has not been tampered with. After successful signature verification, the metadata is decrypted to obtain the file-level metadata and fragment-level metadata of the file content.

[0041] S220. Using a two-layer key derivation algorithm, the fragment-level derivation key and a one-time random number are determined for each fragment file content based on the user password input by the user, the file-level metadata, and the fragment-level metadata.

[0042] Specifically, to verify the user password, the file-level master key is derived by combining the salt value in the file-level metadata with PBKDF2WithHmacSHA256. Then, based on the derived parameters in the fragment-level metadata, the fragment-level derived key and one-time random number for each fragment are recovered.

[0043] S230. Allocate an independent virtual thread for each segment file content, and obtain the segmented encrypted data of each segment file content from the object storage space through the virtual thread. Perform multi-level integrity verification on the segmented encrypted data and the segment-level metadata to obtain the multi-level integrity verification result. The segmented encrypted data is obtained by encrypting the segmented compressed data, and the segmented compressed data is obtained by parallel compression of the content of each segment file using an adaptive dynamic compression algorithm.

[0044] Optionally, multi-level integrity verification is performed on the fragmented encrypted data and the fragment-level metadata to obtain multi-level integrity verification results, including: performing authentication decryption operations on the fragmented encrypted data based on the fragment-level derived key and the one-time random number to obtain a current authentication tag, and comparing the current authentication tag with the authentication tag in the fragment-level metadata to obtain a first-level verification result; performing a hash operation on the fragmented encrypted data to obtain a current fragment hash value, and comparing the current fragment hash value with the fragment hash value in the fragment-level metadata to obtain a second-level verification result; performing digital signature verification on the fragment-level metadata to obtain a third-level verification result; if the first-level verification result, the second-level verification result, and the third-level verification result are all passed, then the multi-level integrity verification result is determined to be passed.

[0045] Specifically, AES-GCM (Advanced Encryption Standard - Galois Counter Mode) authentication tags can be used to verify the integrity of encrypted data, yielding the first-level verification result. Alternatively, fragment hash value verification can be used, comparing the hash value of the downloaded fragment data with the hash value stored in the metadata to obtain the second-level verification result. Finally, metadata signature verification can be used to verify the validity of fragment-level metadata, yielding the third-level verification result. This achieves end-to-end fragment integrity verification, combining AES-GCM authentication tags, fragment hash values, and metadata signatures to form a three-layer protection system. The three levels of verification are progressively enhanced, ensuring integrity from three dimensions: encrypted data, fragment data, and metadata. This avoids vulnerabilities associated with single-verification methods and reduces the risk of data tampering and loss.

[0046] Optionally, if any of the first-level verification result, the second-level verification result, or the third-level verification result fails, an exponential backoff retry mechanism is initiated to trigger the retransmission of the fragmented file content. If the number of retries meets the verification exception condition, the abnormal circuit breaker of the current virtual thread is triggered, and the failed fragment information of the current fragmented file content is recorded.

[0047] The verification failure condition can be that the number of retries exceeds a preset threshold and any one of the first-level verification result, the second-level verification result, or the third-level verification result still fails. The preset threshold can be manually set in advance based on actual conditions or experience. This application embodiment does not specifically limit this; for example, the preset threshold is 3 times.

[0048] S24. If the multi-level integrity verification result is passed, the virtual thread decrypts the encrypted fragment data according to the fragment-level derived key and the one-time random number to obtain the fragment compressed data of each fragment file content. Then, an adaptive dynamic decompression algorithm is used to decompress the fragment compressed data according to the compression level parameters in the fragment-level metadata to obtain the content of each target fragment file.

[0049] Specifically, AES-256-GCM decryption is performed using the fragment-level derived key and unique Nonce of the current fragment file content. After decryption, the appropriate Zstd parameters are used for decompression based on the compression level parameters recorded in the fragment-level metadata to ensure the accuracy and efficiency of decompression.

[0050] S25. Determine the target file based on the content of each target fragment file and the corresponding fragment index.

[0051] Specifically, after each virtual thread completes its task, the main thread can serially merge the contents of each target file segment according to the segment index to obtain the target file. During the serial merging process, only the contents of the currently merged file segment are loaded into memory to ensure constant memory usage. The hash value of the target file can also be compared with the hash value in the file-level metadata. If they match, a successful download result and the target file are returned to the client. Resuming interrupted downloads is also supported; if the download is interrupted, it can resume from the completed segment.

[0052] This application retrieves signed and encrypted metadata from object storage space based on the unique file identifier carried in the file download request, and determines the file-level metadata and fragment-level metadata of the file content based on the signed and encrypted metadata. The signed and encrypted metadata is obtained by digitally signing and encrypting the file-level metadata and fragment-level metadata of the file content. A two-layer key derivation algorithm is used to determine the fragment-level derivation key and a one-time random number for each fragment of the file content based on the user-input password, file-level metadata, and fragment-level metadata. An independent virtual thread is allocated to each fragment of the file content, and the fragment encrypted data of each fragment of the file content is retrieved from object storage space through the virtual thread, and the fragment encrypted data and fragment-level metadata are processed. The system performs multi-level integrity checks to obtain the results. The encrypted fragmented data is obtained by encrypting the compressed fragmented data, which is then compressed in parallel using an adaptive dynamic compression algorithm. If the multi-level integrity checks pass, a virtual thread decrypts the encrypted fragmented data using the fragment-level derived key and a one-time random number to obtain the compressed fragmented data of each file. An adaptive dynamic decompression algorithm is then used to decompress the compressed fragmented data based on the compression level parameters in the fragment-level metadata, yielding the content of each target fragmented file. The target file is then determined based on the content of each target fragmented file and its corresponding fragment index. This scheme employs a two-layer key structure, intelligent dynamic compression, and parallel streaming processing via virtual threads to achieve low-memory, high-concurrency, high-security, and adaptive large file processing, improving the efficiency and security of large file transfers.

[0053] Example 3 Figure 3 This is a schematic diagram of a large file transmission device according to Embodiment 3 of this application. This embodiment is applicable to the intelligent compression and encryption transmission of large files based on virtual threads, particularly suitable for high-concurrency, low-memory, high-security storage and cross-network transmission scenarios of hundreds of gigabytes of ultra-large files. It can be widely used in business scenarios such as large file distribution on video platforms, enterprise-level data backup and synchronization, cloud storage security enhancement, cross-regional large file sharing, and big data and artificial intelligence training data processing. This large file transmission device can be implemented in hardware and / or software and can be configured in computer equipment, such as a server. Figure 3 As shown, the device includes: The fragmentation processing module 310 is used to fragment the file content of the file to be stored according to the file attribute information of the file to be stored and the preset memory threshold, so as to obtain at least two fragmented file contents and corresponding fragment indexes. The segmented compression module 320 is used to allocate independent virtual threads for each segmented file content, and adopts an adaptive dynamic compression algorithm. Based on the file attribute information of each segmented file content, the virtual threads are used to compress each segmented file content in parallel to obtain segmented compressed data of each segmented file content. The data determination module 330 is used to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user password input by the user and the fragment index of each fragment file content using a two-layer key derivation algorithm. The fragment encryption module 340 is used to encrypt the fragment compressed data through the virtual thread using the fragment-level derived key and the one-time random number to obtain the fragment encrypted data of each fragment file content, and upload the fragment encrypted data to the object storage space; The data upload module 450 is used to digitally sign and encrypt the file-level metadata and fragment-level metadata of the file content when all fragment file contents have been successfully uploaded, to obtain signed encrypted metadata, and upload the signed encrypted metadata to the object storage space, and bind it to the fragment encrypted data.

[0054] Optionally, the data determination module 330 includes: The master key and global random number determination unit is used to determine the file-level master key based on the user's input password and a randomly generated salt value using a two-layer key derivation algorithm, and to generate a file-wide random number using a cryptographically secure pseudo-random number generator. The derived key and one-time random number determination unit is used to determine the fragment-level derived key and one-time random number of each fragment file content based on the file-level master key, the file global random number, and the fragment index of each fragment file content.

[0055] Optional, the fragmented compression module 320 includes: The current compression level determination unit is used to determine the current compression level of each file segment based on the type and redundancy of the content of each file segment. The fragmented compressed data determination unit is used to employ an adaptive dynamic compression algorithm to perform parallel compression of the contents of each fragmented file through the virtual thread according to the current compression level, thereby obtaining the fragmented compressed data of each fragmented file content.

[0056] Optionally, the current compression level determination unit is specifically used for: If the current segment file content is of type 1 and the redundancy of the current segment file content is less than the first redundancy threshold, then the current compression level of the current segment file content is determined to be the first compression level; the first type is video or compressed package. If the current fragment file content is of type two, the redundancy of the current fragment file content is greater than or equal to the first redundancy threshold and the redundancy of the current fragment file content is less than the second redundancy threshold, then the current compression level of the current fragment file content is determined to be the second compression level; the second type is different from both the first type and the third type. If the current fragment file content is of type three, and the redundancy of the current fragment file content is greater than or equal to the second redundancy threshold, then the current compression level of the current fragment file content is determined to be the third compression level; the third type is text or document. Wherein, the first redundancy threshold is less than the second redundancy threshold.

[0057] Optionally, the device further includes: The exception handling module is used to initiate an exponential backoff retry mechanism if the processing or upload of the current fragment file content is detected to have failed. If the number of retries meets the exception handling conditions, the module will trigger the circuit breaker of the current virtual thread and record the failed fragment information of the current fragment file content.

[0058] Optionally, the device further includes: The metadata determination module is used to obtain signed and encrypted metadata from the object storage space based on the unique file identifier carried in the file download request, and to determine the file-level metadata and fragment-level metadata of the file content based on the signed and encrypted metadata; wherein, the signed and encrypted metadata is obtained by digitally signing and encrypting the file-level metadata and fragment-level metadata of the file content; The derived key and one-time random number determination module is used to determine the fragment-level derived key and one-time random number of each fragment file content by using a two-layer key derivation algorithm based on the user password input by the user, the file-level metadata, and the fragment-level metadata. The multi-level integrity verification result determination module is used to allocate an independent virtual thread for each fragment file content, obtain the fragmented encrypted data of each fragment file content from the object storage space through the virtual thread, and perform multi-level integrity verification on the fragmented encrypted data and the fragment-level metadata to obtain the multi-level integrity verification result; wherein, the fragmented encrypted data is obtained by encrypting the fragmented compressed data, and the fragmented compressed data is obtained by parallel compression of the content of each fragment file using an adaptive dynamic compression algorithm; The target fragment file content determination module is used to, when the multi-level integrity verification result is passed, decrypt the fragment encrypted data through the virtual thread according to the fragment-level derived key and the one-time random number to obtain the fragment compressed data of each fragment file content, and use an adaptive dynamic decompression algorithm to decompress the fragment compressed data according to the compression level parameters in the fragment-level metadata to obtain the content of each target fragment file; The target file determination module is used to determine the target file based on the content of each target fragment file and the corresponding fragment index.

[0059] Optional, the multi-level integrity verification result determination module includes: The first-level verification result determination unit is used to perform authentication and decryption operations on the fragmented encrypted data according to the fragment-level derived key and the one-time random number to obtain the current authentication tag, and compare the current authentication tag with the authentication tag in the fragment-level metadata to obtain the first-level verification result; The second-level verification result determination unit is used to perform a hash operation on the fragmented encrypted data to obtain the current fragment hash value, and compare the current fragment hash value with the fragment hash value in the fragment-level metadata to obtain the second-level verification result; The third-level verification result determination unit is used to perform digital signature verification on the fragment-level metadata to obtain the third-level verification result. The multi-level integrity verification result determination unit is used to determine that the multi-level integrity verification result is passed if the first-level verification result, the second-level verification result, and the third-level verification result are all passed.

[0060] This application segments the file content of the file to be stored based on its file attribute information and a preset memory threshold, obtaining at least two segments and their corresponding indexes. An independent virtual thread is allocated to each segment, and an adaptive dynamic compression algorithm is used to compress each segment in parallel using the virtual thread, based on its file attribute information, resulting in compressed segment data. A two-layer key derivation algorithm is employed, using the user-input password and the segment index to determine the segment-level derivation key and a one-time random number for each segment. The compressed segment data is then encrypted using the segment-level derivation key and the one-time random number via the virtual thread, resulting in encrypted segment data, which is uploaded to the object storage space. Once all segment data has been successfully uploaded, the file-level metadata and segment-level metadata of the file content are digitally signed and encrypted to obtain signed encrypted metadata, which is then uploaded to the object storage space and bound to the encrypted segment data. The above scheme dynamically adjusts the fragment size based on file attribute information and preset hardware memory thresholds; it employs an adaptive dynamic compression algorithm for parallel intelligent compression, balancing compression ratio and processing speed; it designs a two-layer key structure to improve the security of large file transfers; and it uses a customized virtual thread pool to allocate an independent virtual thread to each fragment, achieving deep binding between the virtual thread and the fragment lifecycle. The creation, execution, and recycling of threads are completely synchronized with the processing lifecycle of fragments. Threads are immediately recycled after fragment processing is completed, and thread circuit breakers are triggered in case of fragment processing exceptions to avoid thread resource leaks. At the same time, the virtual thread pool supports dynamic adjustment of the number of concurrent connections, automatically adapting to memory resources, reducing context switching overhead, and improving the efficiency of large file transfers.

[0061] The large file transfer device provided in this application can execute the large file transfer method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each large file transfer method.

[0062] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0063] Example 4 Figure 4This is a schematic diagram of the structure of an electronic device 410 implementing the large file transfer method of the embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0064] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory 412 or a random access memory 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 412 or loaded from storage unit 418 into the random access memory 413. The random access memory 413 can also store various programs and data required for the operation of the electronic device 410. The processor 411, read-only memory 412, and random access memory 413 are interconnected via a bus 414. An input / output interface 415 is also connected to the bus 414.

[0065] Multiple components in electronic device 410 are connected to input / output interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of monitors, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0066] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as methods for transferring large files.

[0067] In some embodiments, the large file transfer method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 410 via read-only memory 412 and / or communication unit 419. When the computer program is loaded into random access memory 413 and executed by processor 411, one or more steps of the large file transfer method described above can be performed. Alternatively, in other embodiments, processor 411 can be configured for the large file transfer method by any other suitable means (e.g., by means of firmware).

[0068] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0069] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable large file transfer device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0070] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0071] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0072] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0073] A computing system can include requesters and servers. Requesters and servers are generally geographically separated and typically interact via communication networks. The relationship between a requester and a server is established by computer programs running on the respective computers and having a requester-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0074] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0075] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A large file oriented transmission method, characterized by, The method comprises the following steps: According to the file attribute information of the file to be stored and the preset memory threshold, the file content of the file to be stored is fragmented to obtain at least two fragment file contents and corresponding fragment indexes; An independent virtual thread is allocated for each fragment file content, and an adaptive dynamic compression algorithm is used to compress the fragment file contents in parallel through the virtual thread according to the file attribute information of each fragment file content to obtain the fragment compression data of each fragment file content; A double-layer key derivation algorithm is used to determine the fragment-level derived key and the one-time random number of each fragment file content according to the user password input by the user and the fragment index of each fragment file content; The fragment encryption data of each fragment file content is obtained by encrypting the fragment compression data through the virtual thread using the fragment-level derived key and the one-time random number, and the fragment encryption data is uploaded to the object storage space; In the case that all fragment file contents are successfully uploaded, the file-level metadata and the fragment-level metadata of the file content are digitally signed and encrypted to obtain signed and encrypted metadata, and the signed and encrypted metadata is uploaded to the object storage space and is relatedly bound to the fragment encryption data.

2. The method of claim 1, wherein, The double-layer key derivation algorithm is used to determine the fragment-level derived key and the one-time random number of each fragment file content according to the user password input by the user and the fragment index of each fragment file content, which comprises: A double-layer key derivation algorithm is used to determine the file-level master key according to the user password input by the user and the randomly generated salt value, and a cryptography secure pseudo-random number generator is used to generate a file global random number; The file-level master key, the file global random number and the fragment index of each fragment file content are used to determine the fragment-level derived key and the one-time random number of each fragment file content.

3. The method of claim 1, wherein, The adaptive dynamic compression algorithm is used to compress the fragment file contents in parallel through the virtual thread according to the file attribute information of each fragment file content to obtain the fragment compression data of each fragment file content, which comprises: The current compression level of each fragment file content is determined according to the type of each fragment file content and the redundancy of each fragment file content; The adaptive dynamic compression algorithm is used to compress the fragment file contents in parallel through the virtual thread according to the current compression level to obtain the fragment compression data of each fragment file content.

4. The method of claim 3, wherein, The current compression level of each fragment file content is determined according to the type of each fragment file content and the redundancy of each fragment file content, which comprises: If the type of the current fragment file content is the first type and the redundancy of the current fragment file content is less than the first redundancy threshold, the current compression level of the current fragment file content is determined as the first compression level; the first type is a video type or a compressed package type. If the current fragment file content is of type two, the redundancy of the current fragment file content is greater than or equal to the first redundancy threshold and the redundancy of the current fragment file content is less than the second redundancy threshold, then the current compression level of the current fragment file content is determined to be the second compression level; the second type is different from both the first type and the third type. If the current fragment file content is of type three, and the redundancy of the current fragment file content is greater than or equal to the second redundancy threshold, then the current compression level of the current fragment file content is determined to be the third compression level; the third type is text or document. Wherein, the first redundancy threshold is less than the second redundancy threshold.

5. The method of claim 1, wherein, The method further includes: If the processing or upload of the current fragment file content fails, the exponential backoff retry mechanism is initiated. If the number of retries meets the exception handling conditions, the circuit breaker of the current virtual thread is triggered, and the failed fragment information of the current fragment file content is recorded.

6. The method of claim 1, wherein, The method further includes: Based on the unique file identifier carried in the file download request, the signed and encrypted metadata is obtained from the object storage space, and based on the signed and encrypted metadata, the file-level metadata and fragment-level metadata of the file content are determined; wherein, the signed and encrypted metadata is obtained by digitally signing and encrypting the file-level metadata and fragment-level metadata of the file content; A two-layer key derivation algorithm is adopted to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user password input by the user, the file-level metadata, and the fragment-level metadata. An independent virtual thread is allocated to each fragment file content. The virtual thread retrieves the fragmented encrypted data of each fragment file content from the object storage space. Multi-level integrity checks are performed on the fragmented encrypted data and the fragment-level metadata to obtain the multi-level integrity check results. The fragmented encrypted data is obtained by encrypting the fragmented compressed data, and the fragmented compressed data is obtained by parallel compression of the content of each fragment file using an adaptive dynamic compression algorithm. If the multi-level integrity verification result is passed, the virtual thread decrypts the encrypted fragment data according to the fragment-level derived key and the one-time random number to obtain the fragment compressed data of each fragment file content. Then, an adaptive dynamic decompression algorithm is used to decompress the fragment compressed data according to the compression level parameters in the fragment-level metadata to obtain the content of each target fragment file. The target file is determined based on the content of each target fragment file and its corresponding fragment index.

7. The method of claim 6, wherein, Multi-level integrity checks are performed on the fragmented encrypted data and the fragment-level metadata to obtain multi-level integrity check results, including: Based on the fragment-level derived key and the one-time random number, the fragmented encrypted data is subjected to authentication and decryption operations to obtain the current authentication tag. The current authentication tag is then compared with the authentication tag in the fragment-level metadata to obtain the first-level verification result. The hash operation is performed on the fragmented encrypted data to obtain the current fragment hash value, and the current fragment hash value is compared with the fragment hash value in the fragment-level metadata to obtain the second-level verification result; The fragment-level metadata is digitally signed to obtain the third-level verification result; If the first-level verification result, the second-level verification result, and the third-level verification result are all passed, then the multi-level integrity verification result is determined to be passed.

8. A large file oriented transmission apparatus characterized by comprising: include: The sharding module is used to shard the file content of the file to be stored according to the file attribute information and the preset memory threshold, so as to obtain at least two shard file contents and the corresponding shard index. The segmented compression module is used to allocate independent virtual threads for each segment file content. It adopts an adaptive dynamic compression algorithm and performs parallel compression on each segment file content through the virtual threads according to the file attribute information of each segment file content to obtain segmented compressed data of each segment file content. The data determination module is used to determine the fragment-level derivation key and a one-time random number for each fragment file content based on the user's input password and the fragment index of each fragment file content using a two-layer key derivation algorithm. The fragment encryption module is used to encrypt the fragment compressed data through the virtual thread using the fragment-level derived key and the one-time random number to obtain the fragment encrypted data of each fragment file content, and upload the fragment encrypted data to the object storage space; The data upload module is used to digitally sign and encrypt the file-level metadata and fragment-level metadata of the file content after all fragment file contents have been successfully uploaded, to obtain signed encrypted metadata, and upload the signed encrypted metadata to the object storage space, and bind it to the fragment encrypted data.

9. An electronic device, comprising: include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the large file transfer method as described in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method for transferring large files as described in any one of claims 1-7.

Images