Efficient file uploading and downloading method and system

Abstract

The invention discloses an efficient file uploading and downloading method and system, and belongs to the technical field of information technology and cloud computing, the method is implemented by the following steps: preprocessing a file to be transmitted, the preprocessing comprising compressing the file by using a data compression algorithm and extracting file metadata; segmenting the preprocessed file into a plurality of small blocks, and generating verification information for each small block; determining an optimal storage position of each file block by utilizing a distributed hash table technology; the edge computing node assists in executing part of operation tasks to accelerate the transmission process; whether some files or parts of the files are cached on edge nodes or not is intelligently decided according to the user behavior analysis result; when a user needs to download a file, the client firstly checks whether an available cache exists locally, and if not, the client requests the file from the cloud and obtains each file block as required. According to the method, the file uploading and downloading speed can be effectively improved, and the probability of errors in the transmission process is reduced.

Description

Technical Field

[0001] The present invention relates to the field of information technology and cloud computing technology, and in particular to an efficient file uploading and downloading method and system. Background Art

[0002] In the prior art, the uploading and downloading of files mainly rely on the traditional HTTP protocol or FTP service, which has problems such as slow speed and easy errors. Especially when processing large files, these problems are more obvious, affecting the user experience. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide an efficient file upload and download method and system, which can effectively improve the file upload and download rate and reduce the probability of errors occurring during the transmission process.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] An efficient file uploading and downloading method is provided, which realizes file uploading and downloading based on cloud storage. The implementation of the method includes the following steps:

[0006] 1) Preprocessing the files to be transmitted, wherein the preprocessing includes compressing the files using a data compression algorithm and extracting file metadata;

[0007] 2) Split the preprocessed file into multiple small blocks and generate verification information for each small block;

[0008] 3) Use distributed hash table (DHT) technology to determine the best storage location for each file block;

[0009] 4) Use edge computing nodes to assist in executing some computing tasks to speed up the transmission process;

[0010] 5) Select the file to be uploaded when the client is initialized and start the data preprocessing process;

[0011] 6) Upload the file blocks one by one to the designated storage node through the network;

[0012] 7) The receiver immediately verifies the received file block to ensure the integrity of the data and sends a confirmation message to the sender;

[0013] 8) Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis, including the user's access patterns and history records;

[0014] 9) When a user needs to download a file, the client first checks whether there is an available cache locally. If not, it requests the file from the cloud and obtains each file block on demand.

[0015] This method significantly improves file transfer efficiency and reduces network bandwidth consumption by using a specific data compression algorithm and intelligent caching mechanism. This method is particularly suitable for fast exchange scenarios of large-scale data sets.

[0016] Furthermore, the data compression algorithm includes a lossless compression algorithm and a lossy compression algorithm, the lossless compression algorithm includes LZ77 and DEFLATE algorithms; the lossy compression algorithm includes JPEG 2000 algorithm;

[0017] The metadata of the extracted file includes file name, size, and type information.

[0018] Furthermore, in step 2), the size of each small block is automatically adjusted according to the current network bandwidth and delay factors to achieve the best transmission effect;

[0019] The checksum information attached to each file block includes a cyclic redundancy check code CRC or a hash value to ensure data integrity.

[0020] Furthermore, the distributed hash table (DHT) supports decentralized file indexing, quickly locates the location of file blocks, and reduces the pressure on the central server.

[0021] DHT supports decentralized file indexing, and can quickly find the nearest or most suitable copy even if there are multiple copies in the network.

[0022] Furthermore, the edge computing nodes perform simple computing tasks, including preliminary data verification and caching of popular files;

[0023] Edge nodes can act as a temporary buffer to relieve pressure on the main server, especially during peak hours.

[0024] Furthermore, the method also includes an error recovery and retransmission mechanism:

[0025] If a file block fails to be successfully transmitted, the system automatically detects and resends the missing portion;

[0026] Use multi-version concurrency control (MVCC) or other consistency protocols to ensure data consistency.

[0027] Furthermore, the implementation process of this method is as follows:

[0028] S1. Client initialization:

[0029] The user selects the file to upload through the client application;

[0030] The client program starts the data preprocessing process, including compressing the file and generating metadata;

[0031] S2. File segmentation:

[0032] The file is split into multiple blocks, and a checksum is generated for each block;

[0033] The block information is uploaded to the cloud together with the checksum;

[0034] S3, file block positioning:

[0035] Use the DHT query service to determine the best storage location for each file block;

[0036] DHT returns a list of specific addresses where the file blocks should be stored;

[0037] S4. Data transmission:

[0038] File blocks are uploaded one by one to the designated storage node through the network;

[0039] Meanwhile, edge nodes can participate in data reception, especially for files that are expected to be frequently accessed;

[0040] S5. Data verification and confirmation:

[0041] After receiving the file block, the receiver immediately performs a check to ensure the data is complete;

[0042] If the verification fails, a request is made to resend the corresponding file block; if successful, a confirmation message is fed back to the sender;

[0043] S6. Cache management:

[0044] Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis;

[0045] Regularly clean up cached content that is expired or no longer needed;

[0046] S7, File Download:

[0047] When a user needs to download a file, the client first checks whether there is an available cache locally;

[0048] If not, the client requests the file from the cloud and obtains individual file chunks on demand;

[0049] After the file blocks are merged, they are decompressed and restored to the original files and provided to the user.

[0050] The present invention also claims protection for an efficient file uploading and downloading system, comprising:

[0051] The data preprocessing module uses efficient lossless or lossy compression algorithms to preprocess files and extract metadata of files while compressing them;

[0052] Dynamic block module, which dynamically divides files into multiple small blocks according to network conditions and file characteristics;

[0053] Distributed hash table positioning module, which uses DHT technology to assign a unique identifier to each file block and maps it to a specific storage node;

[0054] Edge computing node auxiliary module, deploying edge computing nodes near users, which can perform simple computing tasks;

[0055] Smart caching module, which intelligently determines which files should be cached locally or at edge nodes based on the user's access patterns and history;

[0056] Error recovery and retransmission module, which is used to realize that when a file block fails to be transmitted successfully, the system automatically detects and resends the lost part;

[0057] The system specifically implements file uploading and downloading through the above-mentioned efficient file uploading and downloading method.

[0058] The present invention also claims protection for an efficient file uploading and downloading device, comprising: at least one memory and at least one processor;

[0059] The at least one memory is used to store a machine-readable program;

[0060] The at least one processor is used to call the machine-readable program to implement the above method.

[0061] The present invention also claims protection for a computer-readable medium, characterized in that computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by a processor, the above method is implemented.

[0062] Compared with the prior art, the efficient file uploading and downloading method and system of the present invention has the following beneficial effects:

[0063] The present invention can significantly improve the speed and reliability of file upload and download, while effectively reducing the burden on the data center and providing a better user experience.

[0064] Experiments show that, compared with the traditional method, the present invention can shorten the file upload and download time by at least 30% under the same conditions while maintaining a high success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flowchart of an efficient file uploading and downloading method provided by an embodiment of the present invention;

[0066] Figure 2This is a timing diagram of file upload and download provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below in conjunction with specific embodiments.

[0068] The embodiment of the present invention provides an efficient file upload and download method, which realizes file upload and download based on cloud storage. Figure 1 and Figure 2 As shown, the implementation of this method includes the following steps:

[0069] (1) preprocessing the file to be transmitted, wherein the preprocessing includes compressing the file using a data compression algorithm and extracting file metadata;

[0070] (2) splitting the preprocessed file into multiple small blocks and generating verification information for each small block;

[0071] (3) Use distributed hash table (DHT) technology to determine the best storage location for each file block;

[0072] (4) Use edge computing nodes to assist in executing some computing tasks to speed up the transmission process;

[0073] (5) Select the file to be uploaded when the client is initialized and start the data preprocessing process;

[0074] (6) Upload the file blocks one by one to the designated storage node through the network;

[0075] (7) The receiver immediately verifies the received file block to ensure data integrity and sends a confirmation message to the sender;

[0076] (8) Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis, including the user's access patterns and historical records;

[0077] (9) When a user needs to download a file, the client first checks whether there is an available cache locally. If not, it requests the file from the cloud and obtains each file block on demand.

[0078] The data compression algorithm includes a lossless compression algorithm and a lossy compression algorithm. The lossless compression algorithm includes LZ77 and DEFLATE algorithms; the lossy compression algorithm includes JPEG 2000 algorithm.

[0079] The metadata of the extracted file includes information such as file name, size, type, etc.

[0080] In step (2), the size of each small block is automatically adjusted according to the current network bandwidth and delay factors to achieve the best transmission effect;

[0081] The checksum information attached to each file block includes a cyclic redundancy check code CRC or a hash value to ensure data integrity.

[0082] The distributed hash table (DHT) supports decentralized file indexing, quickly locates the location of file blocks, and reduces the pressure on the central server.

[0083] DHT supports decentralized file indexing, and can quickly find the nearest or most suitable copy even if there are multiple copies in the network.

[0084] The edge computing nodes perform simple computing tasks, including preliminary data verification and caching of popular files;

[0085] Edge nodes can also serve as temporary buffers to relieve pressure on the main servers, especially during peak hours.

[0086] The intelligent caching mechanism determines which files should be cached locally or at edge nodes based on the user's access patterns and history, thereby speeding up the loading of frequently accessed files. The cache update strategy takes into account the freshness of the files and the effective use of storage space.

[0087] The method also includes error recovery and retransmission mechanisms:

[0088] If a file block fails to be successfully transmitted, the system automatically detects and resends the missing portion;

[0089] To ensure data consistency, multi-version concurrency control (MVCC) or other consistency protocols are used.

[0090] The specific implementation process of this method is as follows:

[0091] S1. Client initialization:

[0092] The user selects the file to upload through the client application;

[0093] The client program starts the data preprocessing process, including compressing the files and generating metadata.

[0094] S2. File segmentation:

[0095] The file is split into multiple blocks, and a checksum is generated for each block;

[0096] The block information is uploaded to the cloud together with the checksum.

[0097] S3, file block positioning:

[0098] Use the DHT query service to determine the best storage location for each file block;

[0099] The DHT returns a list of specific addresses where the file blocks should be stored.

[0100] S4. Data transmission:

[0101] File blocks are uploaded one by one to the designated storage node through the network;

[0102] Meanwhile, edge nodes can participate in data reception, especially for files that are expected to be frequently accessed;

[0103] S5. Data verification and confirmation:

[0104] After receiving the file block, the receiver immediately performs a check to ensure the data is complete;

[0105] If the verification fails, a request is made to resend the corresponding file block; if successful, a confirmation message is fed back to the sender.

[0106] S6. Cache management:

[0107] Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis;

[0108] Regularly clean up cached content that is expired or no longer needed.

[0109] S7, File Download:

[0110] When a user needs to download a file, the client first checks whether there is an available cache locally;

[0111] If not, the client requests the file from the cloud and obtains individual file chunks on demand;

[0112] After the file blocks are merged, they are decompressed and restored to the original files and provided to the user.

[0113] This method significantly improves file transfer efficiency and reduces network bandwidth consumption by using a specific data compression algorithm and intelligent caching mechanism. This method is particularly suitable for fast exchange scenarios of large-scale data sets.

[0114] The embodiment of the present invention also provides an efficient file upload and download system, including:

[0115] 1. Data preprocessing module:

[0116] Use efficient lossless or lossy compression algorithms (such as LZ77, DEFLATE, JPEG 2000, etc.) to pre-process files to reduce file size;

[0117] While compressing, the file's metadata (such as file name, size, type, etc.) is extracted to facilitate subsequent management and retrieval.

[0118] 2. Dynamic block module:

[0119] The file is dynamically divided into multiple small blocks according to the network conditions and file characteristics. The size of each small block can be automatically adjusted according to the current network bandwidth, delay and other factors to achieve the best transmission effect.

[0120] Each file block is appended with checksum information (such as CRC or hash value) to ensure data integrity.

[0121] 3. Distributed hash table positioning module:

[0122] DHT technology is used to assign a unique identifier to each file block and map it to a specific storage node; this helps to quickly locate the location of the file block and reduces the pressure on the central server.

[0123] DHT supports decentralized file indexing, and can quickly find the nearest or most suitable copy even if there are multiple copies in the network.

[0124] 4. Edge computing node auxiliary module:

[0125] Edge computing nodes are deployed near users, which can perform simple computing tasks such as preliminary data verification, caching popular files, etc.

[0126] Edge nodes can also serve as temporary buffers to relieve pressure on the main servers, especially during peak hours.

[0127] 5. Intelligent Cache Module:

[0128] Based on the user's access patterns and history, it intelligently determines which files should be cached locally or on edge nodes, thereby speeding up the loading of frequently accessed files.

[0129] The cache update strategy takes into account the freshness of files and the effective use of storage space.

[0130] 6. Error recovery and retransmission module:

[0131] If a block of the file fails to be transferred successfully, the system automatically detects and resends the missing portion.

[0132] To ensure data consistency, multi-version concurrency control (MVCC) or other consistency protocols are used.

[0133] The system specifically implements file upload and download through the efficient file upload and download method described in the above embodiment. The specific process is as follows:

[0134] S1. Client initialization:

[0135] The user selects the file to upload through the client application;

[0136] The client program starts the data preprocessing process, including compressing the files and generating metadata.

[0137] S2. File segmentation:

[0138] The file is split into multiple blocks, and a checksum is generated for each block;

[0139] The block information is uploaded to the cloud together with the checksum.

[0140] S3, file block positioning:

[0141] Use the DHT query service to determine the best storage location for each file block;

[0142] The DHT returns a list of specific addresses where the file blocks should be stored.

[0143] S4. Data transmission:

[0144] File blocks are uploaded one by one to the designated storage node through the network;

[0145] Meanwhile, edge nodes can participate in data reception, especially for files that are expected to be frequently accessed;

[0146] S5. Data verification and confirmation:

[0147] After receiving the file block, the receiver immediately performs a check to ensure the data is complete;

[0148] If the verification fails, a request is made to resend the corresponding file block; if successful, a confirmation message is fed back to the sender.

[0149] S6. Cache management:

[0150] Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis;

[0151] Regularly clean up cached content that is expired or no longer needed.

[0152] S7, File Download:

[0153] When a user needs to download a file, the client first checks whether there is an available cache locally;

[0154] If not, the client requests the file from the cloud and obtains individual file chunks on demand;

[0155] After the file blocks are merged, they are decompressed and restored to the original files and provided to the user.

[0156] The embodiment of the present invention also provides an efficient file uploading and downloading device, comprising: at least one memory and at least one processor;

[0157] The at least one memory is used to store a machine-readable program;

[0158] The at least one processor is used to call the machine-readable program to implement the efficient file uploading and downloading method described in the above embodiment.

[0159] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the efficient file upload and download method described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0160] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0161] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0162] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0163] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0164] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. An efficient file uploading and downloading method, characterized in that: The method for uploading and downloading files based on cloud storage includes the following steps: 1) Preprocessing the files to be transmitted, wherein the preprocessing includes compressing the files using a data compression algorithm and extracting file metadata; 2) Split the preprocessed file into multiple small blocks and generate verification information for each small block; 3) Use distributed hash table technology to determine the best storage location for each file block; 4) Use edge computing nodes to assist in executing some computing tasks to speed up the transmission process; 5) Select the file to be uploaded when the client is initialized and start the data preprocessing process; 6) Upload the file blocks one by one to the designated storage node through the network; 7) The receiver immediately verifies the received file block to ensure the integrity of the data and sends a confirmation message to the sender; 8) Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis; 9) When a user needs to download a file, the client first checks whether there is an available cache locally. If not, it requests the file from the cloud and obtains each file block on demand.

2. According to claim 1, an efficient file upload and download method is characterized in that: The data compression algorithm includes a lossless compression algorithm and a lossy compression algorithm. The lossless compression algorithm includes LZ77 and DEFLATE algorithms; the lossy compression algorithm includes JPEG 2000 algorithm; The metadata of the extracted file includes file name, size, and type information.

3. The efficient file uploading and downloading method according to claim 1, characterized in that: In step 2), the size of each small block is automatically adjusted according to the current network bandwidth and delay factors; The checksum information attached to each file block includes a cyclic redundancy check code or a hash value.

4. The efficient file uploading and downloading method according to claim 1, characterized in that: The distributed hash table supports decentralized file indexing and quickly locates the location of file blocks.

5. The efficient file uploading and downloading method according to claim 1, characterized in that: The edge computing nodes perform simple computing tasks, including preliminary data verification and caching of popular files; Edge nodes can be used as temporary buffers to relieve pressure on the main server.

6. The efficient file uploading and downloading method according to claim 1, characterized in that: The method also includes error recovery and retransmission mechanisms: If a file block fails to be successfully transmitted, the system automatically detects and resends the missing portion; Use multi-version concurrency control or other consistency protocols to ensure data consistency.

7. The efficient file uploading and downloading method according to claim 1, characterized in that: The implementation process of this method is as follows: S1. Client initialization: The user selects the file to upload through the client application; The client program starts the data preprocessing process, including compressing the file and generating metadata; S2. File segmentation: The file is split into multiple blocks, and a checksum is generated for each block; The block information is uploaded to the cloud together with the checksum; S3, file block positioning: Use the DHT query service to determine the best storage location for each file block; DHT returns a list of specific addresses where the file blocks should be stored; S4. Data transmission: File blocks are uploaded one by one to the designated storage node through the network; At the same time, edge nodes can participate in data reception; S5. Data verification and confirmation: After receiving the file block, the receiver immediately verifies it to ensure the data is complete; If the verification fails, a request is made to resend the corresponding file block; if successful, a confirmation message is fed back to the sender; S6. Cache management: Intelligently decide whether to cache certain files or parts of them on edge nodes based on the results of user behavior analysis; Regularly clean up expired or no longer needed cache content; S7, File Download: When a user needs to download a file, the client first checks whether there is an available cache locally; If not, the client requests the file from the cloud and obtains individual file chunks on demand; After the file blocks are merged, they are decompressed and restored to the original files and provided to the user.

8. An efficient file upload and download system, characterized in that: include: The data preprocessing module uses efficient lossless or lossy compression algorithms to preprocess files and extract metadata of files while compressing them; Dynamic block module, which dynamically divides files into multiple small blocks according to network conditions and file characteristics; Distributed hash table positioning module, which uses DHT technology to assign a unique identifier to each file block and maps it to a specific storage node; Edge computing node auxiliary module, deploying edge computing nodes near users, which can perform simple computing tasks; Smart caching module, which intelligently determines which files should be cached locally or at edge nodes based on the user's access patterns and history; Error recovery and retransmission module, which is used to realize that when a file block fails to be transmitted successfully, the system automatically detects and resends the lost part; The system specifically implements file uploading and downloading through the efficient file uploading and downloading method described in any one of claims 1 to 7.

9. An efficient file upload and download device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to implement the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images