A method, system and medium for batch and fast uploading and downloading of a large number of small files

Abstract

The present invention discloses a method, system and medium for batch and rapid uploading and downloading of a large number of small files, belonging to the technical field of computer storage. The method includes: an initialization stage; an upload preprocessing stage, where the files to be uploaded are preprocessed and it is determined whether the conditions for batch uploading are met; an intelligent packaging stage, where intelligent packaging is performed to obtain a packaged body; an upload execution stage, where the packaged body is divided into multiple file blocks, the hash values are matched, and then uploaded to the S3 object storage system; a download initialization stage, when the client receives a file download request from the user, the information of the file to be downloaded specified by the user is obtained and file matching is performed; a download execution stage, for a single file download request, a single file download process is executed; for a batch file download request, a batch file download process is executed. The overall efficiency of data transmission is improved, and the time required for uploading is shortened, especially suitable for large-scale data upload scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of computer storage, and in particular, to a method, system, and medium for batch rapid uploading and downloading of a large number of small files. Background Art

[0002] With the rapid development of information technology, the amount of data has grown explosively, and the processing of a large number of small files has become a thorny problem faced by many fields. In big data analysis scenarios, a large number of small files such as log files and sensor data files need to be frequently uploaded to a data processing platform for analysis and mining; in enterprise-level file management systems, many small files generated during daily office work, such as email attachments and small project documents, also need to be efficiently transmitted and shared within the enterprise network or cloud environment.

[0003] However, traditional file uploading and downloading methods have exposed many drawbacks when faced with a large number of small files. During the uploading process, due to the large number of small files, if each file is uploaded one by one, it will lead to a large amount of network request overhead, and the establishment and disconnection of network connections frequently consume system resources and time, seriously reducing the uploading efficiency. In terms of backend storage and downloading, traditional storage architectures are difficult to handle the high-concurrency read and write requests of a large number of small files. For the download operation, there is a lack of an efficient extraction and transmission strategy for small files. Especially when quickly locating and downloading a single file from a large collection of small files, it often requires traversing the entire storage structure, consuming a large amount of time and unable to meet the user's demand for quick downloading. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, system, and medium for batch rapid uploading and downloading of a large number of small files.

[0005] The purpose of the present invention is achieved through the following technical solutions: In the first aspect of the present invention, there is provided: A method for batch rapid uploading and downloading of a large number of small files, including the following steps:

[0006] In the initialization stage, the client starts the initialization program, checks the system running environment; establishes a secure connection channel with the metadata server and the S3 object storage system, and uses an encrypted communication protocol; loads the client user interface resources and displays the interactive interface for file uploading and downloading operations;

[0007] In the upload preprocessing stage, when the client receives the user's small file upload instruction, it preprocesses the file to be uploaded and determines whether the batch upload condition is met;

[0008] In the intelligent packaging stage, the small files that meet the batch upload condition are intelligently packaged to obtain a package;

[0009] In the upload execution stage, the packaged body is split into multiple file blocks, and after matching the hash values, they are uploaded to the S3 object storage system;

[0010] In the download initialization stage, when the client receives the user's file download request, it obtains the information of the file to be downloaded specified by the user and performs file matching;

[0011] In the download execution stage, for a single-file download request, the single-file download process is executed; for a batch-file download request, the batch-file download process is executed.

[0012] Preferably, the system operating environment includes the operating system version, network connection status, available memory, and disk space; the encryption communication protocol is SSL / TLS; the interaction interface includes a file selection area, an upload progress bar, a download prompt box, and an operation log display area.

[0013] Preferably, the upload preprocessing stage further includes the following steps:

[0014] When the client receives the user's small-file upload instruction, it scans the file path specified by the user or obtains the list of files to be uploaded through a file selection dialog box, and obtains the detailed information of each file, including the file name, file size, file type, creation time, modification time, and the directory path where the file is located;

[0015] Count the total number of files to be uploaded, calculate the total size of all files, analyze the file distribution, including the number of files and the size ratio at different directory depths, and construct a mapping data structure of the file directory structure and file attributes;

[0016] Judge whether the number and size of the files to be uploaded meet the batch upload conditions. If the batch upload conditions are not met, skip the file packaging step and directly enter the single-file upload process; if the batch upload conditions are met, enter the file packaging module for intelligent packaging processing.

[0017] Preferably, the intelligent packaging stage further includes the following steps:

[0018] For small files that meet the batch upload conditions, the file packaging module starts an intelligent calculation algorithm. According to the principle of directory depth-first, it starts screening files from the deepest-level directory, and sets the initial target file number threshold and target total size threshold for the packaged body;

[0019] In the order of file size from small to large, the files are gradually added to the packaged body. Each time a file is added, immediately calculate its start byte position and end byte position in the packaged body, determine the range value, and associate it with the file unique identifier for storage, and construct a local temporary packaged body index data structure;

[0020] During the process of adding files, the current number of files and the total size of the package are monitored in real time. When the preset threshold is reached, the selection of files at the current directory level is paused, and the screening continues in the upper-level directory. This process is repeated until the package is built or all files to be uploaded have been considered;

[0021] If the package still does not meet the threshold requirements after traversing all files, the package is built according to the actually selected files, and the packaging information is recorded;

[0022] Text files are compressed before being added to the package. Image sequence files with similar characteristics are merged or preprocessed before being added to the package, and the packaging algorithm is recorded.

[0023] Preferably, the upload execution stage further includes the following steps:

[0024] After the file packaging is completed, the file upload module divides the package into multiple file blocks according to a predetermined block size, generates multiple file blocks, and assigns a unique serial number and the corresponding hash value calculation task to each file block;

[0025] For each file block, an independent upload task thread is created, an upload task thread pool containing multiple threads is constructed, and the file block upload tasks are assigned to the thread pool for concurrent execution;

[0026] Before each file block is uploaded, its hash value is calculated, and the hash value and the file block data are encapsulated in the upload request and sent to the S3 object storage system; after receiving the upload request, the S3 object storage system first extracts the hash value and performs a hash calculation verification on the file block data. If the two are consistent, it confirms that the file block is received successfully and stores it in the specified location; if not, it sends an error message to the client, requiring the client to re-upload the file block;

[0027] As the file blocks are uploaded, the file upload module updates the upload progress information in real time, including the number of uploaded file blocks, the total number of file blocks, the amount of uploaded data, the total amount of data, and synchronously updates the upload progress information to the progress bar of the user interface on the client side;

[0028] When all file blocks are uploaded and the S3 object storage system returns an upload success message, the file upload module sorts out the metadata information of all files, including file name, original path, file type, range information in the package, compression method; organizes the data according to the predefined data format, and then uploads this metadata information to the metadata server for storage;

[0029] After receiving the metadata information, the metadata server performs data integrity verification and index construction operations, stores the information in the corresponding database table or data structure, and establishes an index relationship between the file metadata and the range information.

[0030] Preferably, the download initialization stage further includes the following steps:

[0031] When the client receives a file download request from the user, it obtains the information of the file to be downloaded specified by the user, including the file name and the file hash value; if the user only provides the file name, the client first searches in the locally cached metadata information. If not found, it sends a query request to the metadata server to obtain the complete metadata information of the file, including the hash value of the package it is in and the range information in the package.

[0032] Preferably, the single-file download process includes the following steps:

[0033] For a single-file download request, the client calculates the range of file blocks to be downloaded based on the obtained file range information and package information, and directly prepares to download the file block according to the recorded range;

[0034] The client creates a download task thread, constructs a single-file download task thread pool, and distributes the file block download tasks to the thread pool for concurrent execution;

[0035] During the download of each file block, the client sends a download request for the package hash value, the file range position, and the signature information to the S3 object storage system. After receiving the request, the S3 object storage system searches for the corresponding file block data based on this information and returns the file block data and the calculated hash value to the client together;

[0036] After receiving the file block, the client first verifies the correctness of the hash value. If it is correct, it assembles the file block data according to the range information of the file in the package, and gradually restores the complete file data; if the hash value is incorrect, it requests the S3 object storage system to re-download the file block;

[0037] As the file blocks are downloaded and assembled, the client updates the download progress information in real time, including the number of downloaded file blocks, the total number of file blocks, the amount of downloaded data, and the total amount of data, and synchronously updates these download progress information to the progress bar of the user interface of the client;

[0038] After all file blocks are downloaded and assembled into a complete file, the client performs a final integrity check on the file. For example, it recalculates the hash value of the file and compares it with the original hash value. If they are the same, it indicates that the file download is successful; if not, error handling is performed according to the situation, including attempting to redownload or prompting the user that the download failed and the reason for the failure.

[0039] Preferably, the batch file download process includes the following steps:

[0040] For a batch file download request, the client calculates the distribution of all files in the package based on the obtained file metadata and range information, and determines the range of all file blocks to be downloaded;

[0041] Allocate corresponding thread numbers according to the number of files, construct a thread pool for batch file download tasks, and distribute the file block download tasks to the thread pool for concurrent execution to perform multi-threaded resume download;

[0042] During the download process of each file block, send a request to the S3 object storage system, receive the file block, verify the hash value, and assemble the file to ensure the correct download of each file block and the complete restoration of the file;

[0043] As the file blocks are downloaded and assembled, the client updates the overall progress information of the batch download in real time, including the number of downloaded files, the total number of files, the amount of downloaded data, and the total amount of data; and displays it on the progress bar of the user interface, and at the same time records the download status and detailed information of each file in the operation log display area;

[0044] When all files are downloaded, the client re-creates the corresponding directory structure on the local disk according to the original directory information in the file metadata, and stores the downloaded files on the disk according to the original directory path to ensure that the organizational structure of the files is the same as when they were originally uploaded;

[0045] During the batch download process, if a network interruption or user pause operation occurs, the client records the current download status information, including the downloaded file block information and the download progress of each file, and stores this information in the local cache; when the network resumes or the user continues the operation, the client reads the download status information from the local cache and continues to execute the download task from the breakpoint;

[0046] If a failure or response delay occurs in the S3 object storage system or the metadata server during the download process, the client automatically activates a fault response mechanism, including increasing the number of retries and the waiting time strategy, and attempts to connect to the server to obtain data multiple times within the first preset time. If the fault duration is greater than the second preset time, the download task is paused and a fault message is prompted to the user. After the server returns to normal, the download is continued from the breakpoint or the download task is restarted according to the user's instruction.

[0047] The second aspect of the present invention provides: A system for batch rapid uploading and downloading of a large number of small files, used to implement any of the above methods for batch rapid uploading and downloading of a large number of small files, including:

[0048] An initialization module, used to start an initialization program using the client, check the system running environment; establish a secure connection channel with the metadata server and the S3 object storage system, and adopt an encrypted communication protocol; load the client user interface resources and display an interactive interface for file uploading and downloading operations;

[0049] An upload preprocessing module, used to preprocess the file to be uploaded after the client receives the user's small file upload instruction, and determine whether the batch upload condition is met;

[0050] An intelligent packaging module, used to intelligently package the small files that meet the batch upload conditions to obtain a package;

[0051] An upload execution module, used to split the package into multiple file blocks, match the hash values, and then upload them to the S3 object storage system;

[0052] A download initialization module, used to obtain the information of the file to be downloaded specified by the user and perform file matching after the client receives the user's file download request;

[0053] A download execution module, used to execute the single file download process for a single file download request; for a batch file download request, execute the batch file download process.

[0054] The third aspect of the present invention provides: A computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, any of the above methods for batch rapid uploading and downloading of a large number of small files is implemented.

[0055] The beneficial effects of the present invention are:

[0056] 1) The intelligent packaging module performs batch processing on small files. According to multi-dimensional factors such as directory depth, file size, and quantity, numerous small files are encapsulated into a packaging body, effectively reducing the number of network requests. For example, thousands of small files originally might require thousands of individual network connection requests, but after packaging, it might only need several requests, greatly reducing the network load and the pressure on the server-side connection processing, improving the overall efficiency of data transmission, shortening the upload time, and being particularly suitable for large-scale data upload scenarios such as enterprise data backup and cloud storage data synchronization business scenarios.

[0057] 2) In terms of downloading, with the help of the file range information stored in the metadata server, accurate positioning and efficient extraction can be achieved for both single-file downloads and batch downloads. For single-file downloads, according to the file hash, range, and signature information, through the object storage segmented download method, only the required part of the file is downloaded, avoiding unnecessary data transmission and greatly accelerating the acquisition speed of a single file. For example, when a user quickly downloads a specific document or picture, it can be obtained immediately. When batch-downloading all small files, multi-threaded breakpoint downloads combine file metadata and range information to regenerate the files and generate them on the disk according to the specified directory, not only ensuring the integrity and accuracy of the download but also making full use of the multi-threaded concurrency advantage to achieve fast batch downloads and improve the overall download efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flowchart of the method for batch and rapid upload and download of a large number of small files;

[0059] Figure 2 is a processing flowchart of the method for batch and rapid upload and download of a large number of small files. DETAILED DESCRIPTION OF THE INVENTION

[0060] Next, the technical solutions of the present invention will be clearly and completely described in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of protection of the present invention.

[0061] First, an explanation of the terms of the present invention: Range information: It refers to specifying the range of the file content to be obtained through a specific request header in an HTTP request. Those skilled in the art can clearly understand its meaning.

[0062] The present invention aims to efficiently achieve batch rapid uploading and downloading operations of a large number of small files. In terms of uploading, through in-depth optimization of the front end, numerous small files are intelligently packaged, and metadata is aggregated and uploaded, effectively reducing the number of network requests. The multi-threaded technology is used to achieve concurrent uploading, and combined with accurate progress display and interruption recovery mechanisms, to ensure the fluency and reliability of the user experience. The back end relies on a powerful distributed object storage system to process and store the uploaded files in parallel. Through the method of segmented downloading Range of object storage, single-file download requests can be quickly responded to, and files can be accurately extracted from the small-file package of distributed storage and transmitted to the user side at high speed. The achievements of this project will be widely applied to many fields such as big data analysis, cloud storage services, and enterprise-level file management, providing strong technical support for the processing of a large number of small files, significantly improving the processing efficiency and user satisfaction of related services, and strongly promoting the digital process and technological innovation and development of the industry.

[0063] Referring to Figure 1 - Figure 2 , the first aspect of the present invention provides: A method for batch rapid uploading and downloading of a large number of small files, including the following steps:

[0064] Initialization stage: The client starts the initialization program to check the system running environment; establishes a secure connection channel with the metadata server and the S3 object storage system, using an encrypted communication protocol; loads the client user interface resources and displays the interactive interface for file uploading and downloading operations;

[0065] Upload preprocessing stage: When the client receives the small file upload instruction from the user, preprocesses the file to be uploaded and determines whether the batch upload condition is met;

[0066] Intelligent packaging stage: Intelligently package the small files that meet the batch upload condition to obtain a package;

[0067] Upload execution stage: Split the package into multiple file blocks, match the hash values, and upload them to the S3 object storage system;

[0068] Download initialization stage: When the client receives the file download request from the user, obtains the information of the file to be downloaded specified by the user and performs file matching;

[0069] Download execution stage: For single-file download requests, execute the single-file download process; for batch file download requests, execute the batch file download process.

[0070] In this embodiment, the client is used to receive the user's small file upload action and trigger file upload; the file packaging module is used to perform intelligent calculations on a batch of small files according to directory depth, file size, number of files, etc., and intelligently package the files into a package according to the set total number and total size, and record the position range of each file in the package; the file upload module is used to concurrently upload the package to the object storage system in a chunked upload manner, and at the same time upload the metadata information of all files and the range corresponding to each file to the metadata server for recording; the object storage system is used for S3 object storage and receives the file storage of the uploaded package; the metadata server is used to store the range information corresponding to the file, and at the same time provide the file range information during the download of a single file; the file download module is used to download the specified content file in a segmented download manner of the object storage according to the obtained file hash, range and signature information.

[0071] For the upload link, in view of the fact that in the traditional method, uploading a large number of small files one by one causes an explosive growth in the number of network requests, leading to serious waste of network resources and low transmission rate problems. The present invention significantly reduces the network request frequency from the root by means of intelligent packaging of numerous small files in the front end and metadata aggregation upload strategy, and at the same time combines multi-threaded concurrent upload technology to fully exploit the potential processing capabilities of network bandwidth and front-end devices, thereby significantly accelerating the upload process.

[0072] In terms of the back-end design, the present invention constructs a powerful distributed object storage system. By using this system to perform parallel processing and storage of uploaded files, it not only enhances the stability and reliability of storage, but also lays a solid foundation for subsequent rapid extraction of files.

[0073] For the single file download requirement, adopting the object storage segmented download Range method can quickly and accurately locate and extract the target file in the small file package of distributed storage, and then transmit it to the user side at high speed, greatly shortening the download waiting time and meeting the user's expectation of quickly obtaining a single file.

[0074] Finally, it can be deeply integrated into key fields such as cloud storage services and enterprise-level file management, provide a solid and efficient technical foundation for the processing of a large number of small files, significantly promote the efficiency leap of related services in processing a large number of small files, and effectively improve the user's satisfaction with related services.

[0075] In some embodiments, the system operating environment includes the operating system version, network connection status, available memory and disk space; the encryption communication protocol is SSL / TLS; the interactive interface includes a file selection area, an upload progress bar, a download prompt box, and an operation log display area.

[0076] In this embodiment, checking the system running environment is to ensure that the basic requirements for small file upload and download operations are met. The SSL / TLS encryption communication protocol is used to ensure the confidentiality, integrity, and authenticity of data transmission, preventing information leakage and malicious tampering. The interactive interface provides users with friendly and intuitive operation guidance and feedback display.

[0077] In some embodiments, the upload preprocessing stage further includes the following steps:

[0078] When the client receives the user's small file upload instruction, it scans the file path specified by the user or obtains the list of files to be uploaded through the file selection dialog box, and obtains the detailed information of each file, including file name, file size, file type, creation time, modification time, and the directory path where the file is located;

[0079] Count the total number of files to be uploaded, calculate the total size of all files, analyze the file distribution, including the number of files and the size ratio at different directory depths, and construct a mapping relationship data structure between the file directory structure and file attributes;

[0080] Judge whether the number and size of the files to be uploaded meet the batch upload conditions. If the batch upload conditions are not met, skip the file packaging step and directly enter the single file upload process; if the batch upload conditions are met, enter the file packaging module for intelligent encapsulation processing.

[0081] In this embodiment, a mapping relationship data structure between the file directory structure and file attributes is constructed for subsequent intelligent processing decisions. Judge whether the number and size of the files to be uploaded meet the preset conditions for the small file batch upload scenario. If there is a single or a small number of files (for example, the number of files is less than or equal to 5 and the total size is less than or equal to 10MB), skip the file packaging step and directly enter the single file upload process; if the batch upload conditions are met, enter the file packaging module for intelligent encapsulation processing.

[0082] In some embodiments, the intelligent packaging stage further includes the following steps:

[0083] For small files that meet the batch upload conditions, the file packaging module starts an intelligent calculation algorithm. According to the directory depth-first principle, files are screened starting from the deepest level directory, and the initial target file number threshold and target total size threshold of the packaging body are set;

[0084] In the order of file size from small to large, files are gradually added to the packaging body. Each time a file is added, immediately calculate its start byte position and end byte position in the packaging body, determine the range value, and associate it with the file unique identifier for storage, constructing a local temporary packaging body index data structure;

[0085] During the process of adding files, the current number of files and the total size of the package are monitored in real time. When the preset threshold is reached, the selection of files in the current directory level is paused, and the screening continues in the upper-level directory. This process is repeated until the package is built or all files to be uploaded have been considered;

[0086] If the package still does not meet the threshold requirements after traversing all files, the package is built according to the actual selected files, and the packaging information is recorded;

[0087] Text files are compressed before being added to the package. Image sequence files with similar characteristics are merged or preprocessed before being added to the package, and the packaging algorithm is recorded.

[0088] In this embodiment, the initial target file number threshold for the package is set to 300, and the target total size threshold is 256MB (these thresholds can be dynamically adjusted according to network bandwidth, server performance, and historical data statistics). For specific types of files (such as highly compressible text files, image sequence files with similar characteristics, etc.), targeted optimization strategies are adopted during packaging. For example, text files are efficiently compressed before being added to the package, and image sequence files are merged or preprocessed to reduce storage space occupancy and transmission data volume, improving the overall packaging and upload efficiency, and the packaging algorithm is recorded.

[0089] In some embodiments, the upload execution stage further includes the following steps:

[0090] After the file packaging is completed, the file upload module divides the package into multiple file blocks according to a predetermined block size, generates a unique sequence number for each file block, and assigns a corresponding hash value calculation task;

[0091] For each file block, an independent upload task thread is created, an upload task thread pool containing multiple threads is constructed, and the file block upload tasks are distributed to the thread pool for concurrent execution;

[0092] Before each file block is uploaded, its hash value is calculated, and the hash value and the file block data are encapsulated in the upload request and sent to the S3 object storage system; after receiving the upload request, the S3 object storage system first extracts the hash value and performs a hash calculation verification on the file block data. If the two are consistent, it confirms that the file block is received successfully and stores it in the specified location; if not, it sends an error message to the client, requesting the client to re-upload the file block;

[0093] As the file blocks are uploaded, the file upload module updates the upload progress information in real time, including the number of uploaded file blocks, the total number of file blocks, the amount of uploaded data, and the total data volume, and synchronously updates the upload progress information to the progress bar of the client's user interface;

[0094] After all file blocks are uploaded and the S3 object storage system returns a successful upload message, the file upload module sorts out the metadata information of all files, including file name, original path, file type, range information in the package, and compression method; organizes the data according to the predefined data format, and then uploads this metadata information to the metadata server for storage;

[0095] After receiving the metadata information, the metadata server performs data integrity verification and index construction operations, stores the information in the corresponding database table or data structure, and establishes an index relationship between the file metadata and the range information.

[0096] In this embodiment, the predetermined block size can be set to 64MB per block or other required sizes. Establishing an efficient index relationship between the file metadata and the range information can facilitate quick querying and positioning during subsequent file downloading and management.

[0097] In some embodiments, the download initialization stage further includes the following steps:

[0098] When the client receives the user's file download request, it obtains the information of the file to be downloaded specified by the user, including the file name and file hash value; if the user only provides the file name, the client first searches in the locally cached metadata information, and if not found, it sends a query request to the metadata server to obtain the complete metadata information of the file, including the hash value of the package it is in and the range information in the package.

[0099] In some embodiments, the single-file download process includes the following steps:

[0100] For a single-file download request, the client calculates the range of file blocks to be downloaded based on the obtained file range information and package information, and directly prepares to download the file block according to the recorded range;

[0101] The client creates a download task thread, constructs a single-file download task thread pool, and distributes the file block download tasks to the thread pool for concurrent execution;

[0102] During the download process of each file block, the client sends a download request for the package hash value, file range position, and signature information to the S3 object storage system. After receiving the request, the S3 object storage system searches for the corresponding file block data based on this information and returns the file block data together with the calculated hash value to the client;

[0103] After the client receives a file block, it first verifies the correctness of the hash value. If it is correct, it assembles the file block data according to the range information of the file in the package, gradually restoring the complete file data; if the hash value is incorrect, it requests the S3 object storage system to re-download the file block.

[0104] As the file blocks are downloaded and assembled, the client updates the download progress information in real time, including the number of downloaded file blocks, the total number of file blocks, the amount of downloaded data, and the total amount of data, and synchronously updates these download progress information to the progress bar of the user interface of the client.

[0105] When all file blocks are downloaded and assembled into a complete file, the client performs a final integrity check on the file, such as calculating the hash value of the file again and comparing it with the original hash value. If they are the same, it means the file is downloaded successfully; if they are different, error handling is performed according to the situation, including attempting to re-download or prompting the user that the download fails and the reason for the failure.

[0106] In some embodiments, the batch file download process includes the following steps:

[0107] For a batch file download request, the client calculates the distribution of all files in the package based on the obtained file metadata and range information, and determines the ranges of all file blocks to be downloaded.

[0108] Allocate the corresponding number of threads according to the number of files, construct a thread pool for batch file download tasks, and allocate the file block download tasks to the thread pool for concurrent execution to perform multi-threaded resume download.

[0109] During the download process of each file block, send a request to the S3 object storage system, receive the file block, verify the hash value, and assemble the file to ensure the correct download of each file block and the complete restoration of the file.

[0110] As the file blocks are downloaded and assembled, the client updates the overall progress information of the batch download in real time, including the number of downloaded files, the total number of files, the amount of downloaded data, and the total amount of data; and displays it on the progress bar of the user interface, and at the same time records the download status and detailed information of each file in the operation log display area.

[0111] When all files are downloaded, the client re-creates the corresponding directory structure on the local disk according to the original directory information in the file metadata, and stores the downloaded files on the disk according to the original directory path to ensure that the organizational structure of the files is the same as when they were originally uploaded.

[0112] During the batch download process, if a network interruption occurs or the user pauses the operation, the client records the current download status information, including the downloaded file block information and the download progress of each file, and stores this information in the local cache; when the network resumes or the user continues the operation, the client reads the download status information from the local cache and continues to execute the download task from the breakpoint.

[0113] If a failure or response delay occurs in the S3 object storage system or the metadata server during the download process, the client automatically activates a fault response mechanism, including increasing the number of retries and the waiting time strategy, and attempts to connect to the server to obtain data multiple times within the first preset time; if the failure duration is greater than the second preset time, the download task is paused and a fault message is prompted to the user. After the server returns to normal, the download continues from the breakpoint or the download task is restarted according to the user's instruction.

[0114] Intelligent file packaging and indexing technology: The present invention deeply analyzes the characteristics of batch small files, such as directory depth, size, and quantity, and uses intelligent computing algorithms to encapsulate them into a package. During the packaging process, the position range of each file in the package is accurately recorded, and an efficient local temporary index is constructed. This technology effectively reduces the number of network requests and network overhead, and at the same time provides a fast and accurate data positioning basis for subsequent file upload, download, and management operations, greatly improving the overall efficiency and data manageability in processing a large number of small files.

[0115] Multi-threaded concurrent upload and download mechanism: In the upload process, the present invention divides the package into blocks and uploads them to the object storage system using multi-threaded concurrency, making full use of network bandwidth resources to achieve high-speed data transmission. At the same time, during download, according to the file hash, range, and signature information, threads are intelligently allocated according to the number of files to be downloaded. For single-file download, a segmented download method is adopted, and for batch download, a multi-threaded breakpoint download strategy is used. This mechanism significantly improves the speed and stability of upload and download, effectively meets the high-concurrency requirements in the scenario of transmitting a large number of small files, and ensures the smoothness and timeliness of data transmission.

[0116] Metadata server and object storage collaboration technology: The metadata server is responsible for storing the metadata information and range information of files, and can quickly provide key data guidance during file download. The S3 object storage focuses on receiving and storing the uploaded package files. The two work together to achieve the separate management of data storage and index information, ensuring both the efficiency and security of file storage, and being able to quickly locate the target file among a large number of small files through accurate metadata indexing. Whether it is a single-file or batch-file operation, it can respond efficiently, improving the reliability and performance of the entire system in processing a large number of small files.

[0117] The second aspect of the present invention provides: A system for batch and rapid uploading and downloading of a large number of small files, which is used to implement any of the above methods for batch and rapid uploading and downloading of a large number of small files, including:

[0118] An initialization module, which is used to start an initialization program using a client, check the system running environment; establish a secure connection channel with a metadata server and an S3 object storage system, and adopt an encrypted communication protocol; load client user interface resources and display an interactive interface for file uploading and downloading operations;

[0119] An upload preprocessing module, which is used to preprocess the file to be uploaded after the client receives the small file upload instruction from the user, and determine whether the batch upload condition is met;

[0120] An intelligent packaging module, which is used to intelligently package the small files that meet the batch upload conditions to obtain a package;

[0121] An upload execution module, which is used to split the package into multiple file blocks, match the hash values, and then upload them to the S3 object storage system;

[0122] A download initialization module, which is used to obtain the information of the file to be downloaded specified by the user and perform file matching after the client receives the user's file download request;

[0123] A download execution module, which is used to execute a single file download process for a single file download request; for a batch file download request, execute a batch file download process.

[0124] The third aspect of the present invention provides: A computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, any of the above methods for batch and rapid uploading and downloading of a large number of small files is implemented.

[0125] The above is only the preferred implementation manner of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications, and environments, and can be changed within the scope of the concept described herein through the above teachings or the technology or knowledge in related fields. And the changes and modifications made by those skilled in the art that do not depart from the spirit and scope of the present invention should all be within the protection scope of the appended claims of the present invention.

Claims

1. A method for batch and rapid uploading and downloading of a large number of small files, characterized in that: It includes the following steps: In the initialization stage, the client starts the initialization program to check the system running environment; establishes a secure connection channel with the metadata server and the S3 object storage system, and adopts an encrypted communication protocol; loads the client user interface resources and displays the interactive interface for file upload and download operations; In the upload preprocessing stage, when the client receives the small file upload instruction from the user, it preprocesses the file to be uploaded and determines whether the batch upload condition is met; In the intelligent packaging stage, the small files that meet the batch upload condition are intelligently packaged to obtain a package; In the upload execution stage, the package is split into multiple file blocks, the hash values are matched, and then uploaded to the S3 object storage system; In the download initialization stage, when the client receives the file download request from the user, it obtains the information of the file to be downloaded specified by the user and performs file matching; In the download execution stage, for a single file download request, the single file download process is executed; for a batch file download request, the batch file download process is executed; The intelligent packaging stage further includes the following steps: For the small files that meet the batch upload condition, the file packaging module starts the intelligent calculation algorithm. According to the directory depth-first principle, it starts screening files from the deepest-level directory, and sets the initial target file number threshold and target total size threshold for the package; In the order of file size from small to large, the files are gradually added to the package. For each added file, its start byte position and end byte position in the package are immediately calculated, the range value is determined, and it is associated with the file unique identifier for storage, and the local temporary package index data structure is constructed; During the process of adding files, the current file number and total size of the package are monitored in real time. When the preset threshold is reached, the file selection in the current directory level is paused, and the screening continues in the upper-level directory. Repeat this process until the package is constructed or all files to be uploaded have been considered; If after traversing all files, the package still does not meet the threshold requirements, the package is constructed according to the actual selected files, and the packaging information is recorded; The text files are compressed and then added to the package. The image sequence files with similar characteristics are merged or preprocessed and then added to the package, and the packaging algorithm is recorded.

2. The method for batch and rapid uploading and downloading of a large number of small files according to claim 1, wherein: The system running environment includes the operating system version, network connection status, available memory and disk space; the encrypted communication protocol is SSL / TLS; the interactive interface includes a file selection area, an upload progress bar, a download prompt box, and an operation log display area.

3. The method for batch fast uploading and downloading of a large number of small files according to claim 1, wherein: The upload preprocessing stage further includes the following steps: When the client receives the small file upload instruction from the user, it scans the file path specified by the user or obtains the list of files to be uploaded through the file selection dialog box, and obtains the detailed information of each file, including the file name, file size, file type, creation time, modification time, and the directory path where the file is located; Counts the total number of files to be uploaded, calculates the total size of all files, analyzes the file distribution, including the number of files and the size proportion under different directory depths, and constructs the mapping relationship data structure between the file directory structure and file attributes; Judge whether the quantity and size of the files to be uploaded meet the conditions for batch upload. If the conditions for batch upload are not met, skip the file packaging step and directly enter the single-file upload process; if the conditions for batch upload are met, enter the file packaging module for intelligent encapsulation processing.

4. The method for batch fast uploading and downloading of a large number of small files according to claim 1, characterized in that: The upload execution stage also includes the following steps: After the file packaging is completed, the file upload module divides the package body according to a predetermined chunk size to generate multiple file chunks, and assigns a unique serial number and a corresponding hash value calculation task to each file chunk; For each file chunk, create an independent upload task thread, construct an upload task thread pool containing multiple threads, and allocate the file chunk upload tasks to the thread pool for concurrent execution; Before each file chunk is uploaded, calculate its hash value, encapsulate the hash value and the file chunk data together in the upload request and send it to the S3 object storage system; after receiving the upload request, the S3 object storage system first extracts the hash value and performs a hash calculation verification on the file chunk data. If the two are consistent, it confirms that the file chunk is successfully received and stores it in the specified location; if they are inconsistent, it sends an error message to the client, asking the client to re-upload the file chunk; As the file chunks are uploaded, the file upload module updates the upload progress information in real time, including the number of uploaded file chunks, the total number of file chunks, the amount of uploaded data, and the total amount of data, and synchronously updates the upload progress information to the progress bar of the user interface on the client side; When all file chunks are uploaded and the S3 object storage system returns an upload success message, the file upload module sorts out the metadata information of all files, including file name, original path, file type, range information in the package body, and compression method; organizes the data according to the predefined data format, and then uploads this metadata information to the metadata server for storage; After receiving the metadata information, the metadata server performs data integrity verification and index construction operations, stores the information in the corresponding database table or data structure, and establishes an index relationship between the file metadata and the range information.

5. The method for batch fast uploading and downloading of a large number of small files according to claim 1, characterized in that: The download initialization stage also includes the following steps: When the client receives the user's file download request, obtain the information of the file to be downloaded specified by the user, including file name and file hash value; if the user only provides the file name, the client first searches in the locally cached metadata information, and if not found, sends a query request to the metadata server to obtain the complete metadata information of the file, including the hash value of the package body where it is located and the range information in the package body.

6. The method for batch fast uploading and downloading of a large number of small files according to claim 1, wherein: The single-file download process includes the following steps: For a single-file download request, the client calculates the range of file chunks to be downloaded according to the obtained file range information and package body information, and directly prepares to download the file chunk according to the recorded range; The client creates a download task thread, constructs a single-file download task thread pool, and allocates the file chunk download tasks to the thread pool for concurrent execution; During the download process of each file block, the client sends a download request for the package hash value, the file range location, and the signature information to the S3 object storage system. After receiving the request, the S3 object storage system searches for the corresponding file block data based on this information and returns the file block data along with the calculated hash value to the client; After receiving the file block, the client first verifies the correctness of the hash value. If it is correct, the client assembles the file block data according to the range information of the file in the package, gradually restoring the complete file data; if the hash value is incorrect, the client requests the S3 object storage system to re-download the file block; As the file blocks are downloaded and assembled, the client updates the download progress information in real time, including the number of downloaded file blocks, the total number of file blocks, the amount of downloaded data, and the total amount of data, and synchronously updates this download progress information to the progress bar of the user interface of the client; When all file blocks are downloaded and assembled into a complete file, the client performs a final integrity check on the file, such as calculating the hash value of the file again and comparing it with the original hash value. If they are consistent, it means the file is downloaded successfully; if they are inconsistent, error handling is performed according to the situation, including attempting to re-download or prompting the user that the download fails and the reason for the failure.

7. The method for batch and rapid uploading and downloading of a large number of small files according to claim 1, wherein: The described batch file download process includes the following steps: For a batch file download request, the client calculates the distribution of all files in the package based on the obtained file metadata and range information, and determines the range of all file blocks to be downloaded; Allocate the corresponding number of threads according to the number of files, construct a thread pool for batch file download tasks, and allocate the file block download tasks to the thread pool for concurrent execution to perform multi-threaded resume download; During the download process of each file block, send a request to the S3 object storage system, receive the file block, verify the hash value, and assemble the file to ensure the correct download of each file block and the complete restoration of the file; As the file blocks are downloaded and assembled, the client updates the overall progress information of the batch download in real time, including the number of downloaded files, the total number of files, the amount of downloaded data, and the total amount of data; and displays it on the progress bar of the user interface, and at the same time records the download status and detailed information of each file in the operation log display area; When all files are downloaded, the client re-creates the corresponding directory structure on the local disk according to the original directory information in the file metadata, and stores the downloaded files on the disk according to the original directory path to ensure that the organizational structure of the files is the same as when they were originally uploaded; During the batch download process, if a network interruption or user pause operation occurs, the client records the current download status information, including the downloaded file block information and the download progress of each file, and stores this information in the local cache; when the network resumes or the user continues to operate, the client reads the download status information from the local cache and continues to execute the download task from the breakpoint; If, during the download process, the S3 object storage system or the metadata server fails or experiences a response delay, the client automatically activates a fault response mechanism, including increasing the number of retries and the waiting time strategy, and attempts to connect to the server to obtain data multiple times within the first preset time; if the fault duration is greater than the second preset time, the download task is paused and a fault message is prompted to the user. After the server resumes normal operation, the download is continued from the breakpoint or the download task is restarted according to the user's instruction.

8. A system for batch and rapid uploading and downloading of a large number of small files, characterized in that: A method for implementing the batch fast upload and download of a large number of small files as described in any one of claims 1-7, including: An initialization module, used to start an initialization program using the client, check the system running environment; establish a secure connection channel with the metadata server and the S3 object storage system, and adopt an encrypted communication protocol; load the client user interface resources and display an interactive interface for file upload and download operations; An upload preprocessing module, used to preprocess the file to be uploaded after the client receives the user's small file upload instruction, and determine whether the batch upload condition is met; An intelligent packaging module, used to intelligently package the small files that meet the batch upload conditions to obtain a package; An upload execution module, used to split the package into multiple file blocks, match the hash values, and then upload them to the S3 object storage system; A download initialization module, used to obtain the information of the file to be downloaded specified by the user and perform file matching after the client receives the user's file download request; A download execution module, used to execute the single-file download process for a single-file download request; for a batch file download request, execute the batch file download process; The intelligent packaging module is further used to start an intelligent calculation algorithm for the small files that meet the batch upload conditions, screen the files starting from the deepest-level directory according to the directory depth-first principle, and set the initial target file quantity threshold and target total size threshold of the package; Add the files to the package step by step in ascending order of file size. For each added file, immediately calculate its start byte position and end byte position in the package, determine the range value, and associate it with the file unique identifier for storage, and construct a local temporary package index data structure; During the process of adding files, monitor the current file quantity and total size of the package in real time. When the preset threshold is reached, pause the file selection at the current directory level, and then continue to screen the upper-level directory. Repeat this process until the package is constructed or all files to be uploaded have been considered; If, after traversing all files, the package still does not meet the threshold requirements, complete the construction of the package according to the actual selected files and record the packaging information; Compress the text files and then add them to the package, merge or preprocess the image sequence files with similar characteristics and then add them to the package, and record the packaging algorithm.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the method for batch fast upload and download of a large number of small files as described in any one of claims 1-7 is implemented.

Images