A multi-level cache and file batch downloading method, system and storage medium

Abstract

The present invention discloses a multi-level cache and file batch download method, system and storage medium, which belongs to the field of computer storage technology. The method includes: S1: the object storage gateway kernel loads the file list, and assembles and deduplicates the file list; S2: setting the file upload threshold, grouping small files, and sharding large files; S3: using the number of small file grouping threads plus the number of large file sharding threads to calculate the number of threads that need to be opened by the cloud storage gateway; batch opening multiple threads for file download; S4: after the cloud storage service end obtains the file, it caches it according to the first heat threshold. It has stronger reliability and adopts the NDN+IP-based C / S system structure. Even if the central server fails, the customer business system will not be affected at all when the edge network connection is normal. It has stronger security and the files are uploaded using block-by-block deduplication and encryption. Compared with traditional object storage, it greatly reduces the risk of data leakage.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a multi-level cache and file batch downloading method, system and storage medium. Background Art

[0002] The rapid development of cloud computing has led to an increasing number of users storing and accessing data in the cloud. However, due to current issues with cloud storage, such as security, availability, convenience, and access speed, traditional applications struggle to integrate well with cloud storage. Most public cloud providers rely on the Internet Protocol (IP), typically using REST APIs over HTTP. While this is useful for creating new applications, it imposes significant costs on compatibility and adaptation for existing systems. Due to incompatibilities between public cloud technologies and existing applications, object storage gateways are needed between cloud storage systems and enterprise applications. Object storage gateways require local caching, data security, and ease of use. They offer competitive pricing compared to traditional storage and can help enterprises easily migrate to the cloud. Object storage gateways provide basic protocol conversion and simple connectivity, enabling transparent communication between incompatible technologies. Gateways can make object storage appear like NAS devices, block storage arrays, local disks, backup targets, or even extensions of the application itself.

[0003] Currently, users experience inefficiencies when reading and storing data using specific cloud objects (such as Tianyi Cloud). This problem is primarily due to the fact that cloud storage utilizes centralized servers, requiring interaction with centralized servers at the center of the network to satisfy relevant requirements. Specifically, the pipeline is single-source and single-path, which can easily cause congestion. For example, the same video may need to be sent numerous times from a single central server. Second, failures in business systems and object storage networks can severely impact the normal operation of these systems. Therefore, the present invention proposes a communication method for efficient storage that addresses these issues with minimal modifications to existing systems. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a multi-level caching and file batch downloading method, system and storage medium.

[0005] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides: a multi-level cache and file batch downloading method, comprising the following steps:

[0006] S1: The object storage gateway kernel loads the file list, assembles the file list, and deduplicates it.

[0007] S2: Set the file upload threshold. Files smaller than the file upload threshold are considered small files, and files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented.

[0008] S3: Calculate the number of threads required for the cloud storage gateway by adding the number of threads for small file grouping to the number of threads for large file sharding; then enable multiple threads in batches for file downloads.

[0009] S4: After the cloud storage service obtains the file, it caches it according to the first heat threshold.

[0010] Preferably, the S1 further comprises the following steps:

[0011] The object storage gateway kernel loads the file list and mounts the local Posix disk through the object storage gateway. The object storage gateway schedules with the operating system kernel to obtain the file list, including the file's corresponding hash value, file size, and file attributes, and initiates file download.

[0012] Assemble and deduplicate the file list, calculate the total number of files, perform integration analysis based on the file hash value, analyze the file duplication, and determine whether there are duplicate files; if there are duplicate file hash values, remove the duplicate files from the download list and record the duplicate file information locally.

[0013] Preferably, the S2 further comprises the following steps:

[0014] First, the small files are grouped according to the number of files and the file upload threshold. Then, the file list is looped through and the file sizes are added up to calculate the group size.

[0015] When the target download file is a large file, the download fragmentation rule is calculated. The number of download fragments = target download file size / file download threshold. The number of groups is increased by 1, and the group size is recalculated.

[0016] When the target upload file is a large file, the upload fragmentation rule is calculated. The number of upload fragments = the sum of the target upload file sizes / the file upload threshold. When the size of this batch of uploaded files is larger than the file upload threshold, the number of groups is increased by 1.

[0017] Calculate the overall group size through the loop.

[0018] Preferably, the step S3 further comprises the following steps:

[0019] After calculating the number of threads required to be opened by the cloud storage gateway, batch-enable multiple threads for file downloading based on the number of threads that can be opened by the storage gateway client.

[0020] The upload thread batch checks whether there are files with corresponding hash values ​​in the local disk cache based on the file hash values. If the files exist in the local disk cache, the local IO pair is used to open the files directly, and the cloud storage gateway returns the file stream to the application through the system kernel. If the files do not exist in the local disk cache, the storage gateway initiates a batch file download request to the cloud storage gateway server.

[0021] When the cloud storage service receives a batch download request, it will perform a cyclic check on the download file list; compare the hash value of the file in the download file list with the file in the Redis cache; if the file exists, the cloud storage service will read the file stream from the high-speed temporary buffer for output, record the number of file downloads and download time, and write them to the Redis cache; if the file does not exist, the cloud storage service will initiate a file download request to the object storage gateway; based on its own business logic, the object storage gateway obtains the file storage location through the crash algorithm, accelerates it through the SSD cache, and returns the file stream for the cloud storage service to download.

[0022] Preferably, the S4 further comprises the following steps:

[0023] After the cloud storage service obtains the file, it determines whether it needs to be cached based on the locally recorded file information; if the heat value is higher than the first heat threshold, the file is downloaded to the storage gateway cache area; if the heat value is lower than the first heat threshold, the file is directly input to the storage gateway client through the proxy method;

[0024] Based on the returned file streams, the storage gateway server combines all file streams into a packaged file stream according to the file size range information; the returned file metadata information records the total data size of the packaged file stream, the size of each file in it, and the corresponding order of the files;

[0025] The storage gateway client obtains the packaged file stream, parses it, and then downloads it to the local disk. The local disk starts multi-threading based on the number of files and splits the stream file according to the file metadata size, file sequence, and pointer position.

[0026] Restore each split file according to the corresponding metadata information and dispatch it to the application flow through the kernel;

[0027] The local disk records the file's corresponding hash value, file download count, and file download time;

[0028] The local cache detection process calculates the file heat value based on the number of file downloads, file call times and file download time;

[0029] According to the set local cache space size and the file heat value sorting of each file, cache files whose file heat value is lower than the second heat threshold are asynchronously deleted.

[0030] A second aspect of the present invention provides: a multi-level cache and file batch downloading system, for implementing any of the above-mentioned multi-level cache and file batch downloading methods, comprising:

[0031] The loading module is used to load the file list using the object storage gateway kernel, and assemble and deduplicate the file list;

[0032] A processing module is used to set a file upload threshold. Files smaller than the file upload threshold are considered small files, while files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented.

[0033] The download module is used to calculate the number of threads required by the cloud storage gateway by adding the number of threads for small file grouping and the number of threads for large file sharding; and to batch start multiple threads for file downloading;

[0034] The cache module is used to cache files according to a first heat threshold after obtaining the files from the cloud storage service end.

[0035] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned multi-level caching and file batch downloading methods is implemented.

[0036] The beneficial effects of the present invention are:

[0037] 1) Greater reliability: Using an NDN+IP-based C / S system structure, even if the central server fails, the customer's business system will not be affected at all as long as the edge network connection is normal.

[0038] 2) Stronger security: Files are uploaded using block-based deduplication and encryption, which greatly reduces the risk of data leakage compared to traditional object storage.

[0039] 3) Faster read and write speeds. Based on a design that caches hot and cold data locally, read and write speeds rely on local cache devices. This also reduces the concurrency pressure on Tianyi Cloud Object Storage. The object storage gateway's caching design reduces the interaction pressure between clients and object storage to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the multi-level cache and file batch downloading method;

[0041] Figure 2Detailed flow chart of multi-level caching and batch downloading method;

[0042] Figure 3 Flowchart for downloading large files in pieces;

[0043] Figure 4 Flowchart for group downloading of small files;

[0044] Figure 5 Calculate the flow chart for cache file cleaning. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0046] This method utilizes the NDN (Named Data Network) and Posix-based client / server (C / S) architecture to achieve conversion between the two protocols. Based on the Posix C / S architecture, this method improves the responsiveness of cloud storage services through mechanisms such as local data grouping and sharding, multi-level caching, and concurrent uploads. By integrating cloud storage and the Internet, this method establishes a low-cost, scalable, secure, and convenient integrated cloud storage platform that combines local high-speed caching and cloud storage.

[0047] See Figure 1-Figure 5 The first aspect of the present invention provides: a multi-level cache and file batch downloading method, comprising the following steps:

[0048] S1: The object storage gateway kernel loads the file list, assembles the file list, and deduplicates it.

[0049] S2: Set the file upload threshold. Files smaller than the file upload threshold are considered small files, and files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented.

[0050] S3: Calculate the number of threads required by the cloud storage gateway by adding the number of threads for small file grouping to the number of threads for large file sharding; then enable multiple threads in batches for file downloads.

[0051] S4: After the cloud storage service obtains the file, it caches it according to the first heat threshold.

[0052] In this embodiment, the client utilizes an IP-based client / server architecture, utilizing a local data caching mechanism to improve the response speed of the cloud storage service. Through hardware and software configuration, high-speed local network storage is achieved. A three-level cache system, comprising gateway local storage, gateway server storage, and object storage components, accelerates downloads through multi-level caching. Each cache level calculates a popularity value based on file download time, number of downloads, and number of file calls. The object storage gateway utilizes an NDN (Name Data Network) + IP-based client / server architecture, implementing protocol conversion at the gateway. Using NDN network storage as the backend, the object storage gateway uploads encrypted data to the NDN network storage.

[0053] In some embodiments, the S1 further comprises the following steps:

[0054] The object storage gateway kernel loads the file list and mounts the local Posix disk through the object storage gateway. The object storage gateway schedules with the operating system kernel to obtain the file list, including the file's corresponding hash value, file size, and file attributes, and initiates file download.

[0055] Assemble and deduplicate the file list, calculate the total number of files, perform integration analysis based on the file hash value, analyze the file duplication, and determine whether there are duplicate files; if there are duplicate file hash values, remove the duplicate files from the download list and record the duplicate file information locally.

[0056] In some embodiments, the S2 further comprises the following steps:

[0057] First, the small files are grouped according to the number of files and the file upload threshold. Then, the file list is looped through and the file sizes are added up to calculate the group size.

[0058] When the target download file is a large file, the download fragmentation rule is calculated. The number of download fragments = target download file size / file download threshold. The number of groups is increased by 1, and the group size is recalculated.

[0059] When the target upload file is a large file, the upload fragmentation rule is calculated. The number of upload fragments = the sum of the target upload file sizes / the file upload threshold. When the size of this batch of uploaded files is larger than the file upload threshold, the number of groups is increased by 1.

[0060] Calculate the overall group size through the loop.

[0061] In some embodiments, the S3 further comprises the following steps:

[0062] After calculating the number of threads required to be opened by the cloud storage gateway, batch-enable multiple threads for file downloading based on the number of threads that can be opened by the storage gateway client.

[0063] The upload thread batch checks whether there are files with corresponding hash values ​​in the local disk cache based on the file hash values. If the files exist in the local disk cache, the local IO pair is used to open the files directly, and the cloud storage gateway returns the file stream to the application through the system kernel. If the files do not exist in the local disk cache, the storage gateway initiates a batch file download request to the cloud storage gateway server.

[0064] When the cloud storage service receives a batch download request, it will perform a cyclic check on the download file list; compare the hash value of the file in the download file list with the file in the Redis cache; if the file exists, the cloud storage service will read the file stream from the high-speed temporary buffer for output, record the number of file downloads and download time, and write them to the Redis cache; if the file does not exist, the cloud storage service will initiate a file download request to the object storage gateway; based on its own business logic, the object storage gateway obtains the file storage location through the crash algorithm, accelerates it through the SSD cache, and returns the file stream for the cloud storage service to download.

[0065] In some embodiments, the S4 further comprises the following steps:

[0066] After the cloud storage service obtains the file, it determines whether it needs to be cached based on the locally recorded file information; if the heat value is higher than the first heat threshold, the file is downloaded to the storage gateway cache area; if the heat value is lower than the first heat threshold, the file is directly input to the storage gateway client through the proxy method;

[0067] Based on the returned file streams, the storage gateway server combines all file streams into a packaged file stream according to the file size range information; the returned file metadata information records the total data size of the packaged file stream, the size of each file in it, and the corresponding order of the files;

[0068] The storage gateway client obtains the packaged file stream, parses it, and then downloads it to the local disk. The local disk starts multi-threading based on the number of files and splits the stream file according to the file metadata size, file sequence, and pointer position.

[0069] Restore each split file according to the corresponding metadata information and dispatch it to the application flow through the kernel;

[0070] The local disk records the file's corresponding hash value, file download count, and file download time;

[0071] The local cache detection process calculates the file heat value based on the number of file downloads, file call times and file download time;

[0072] According to the set local cache space size and the file heat value sorting of each file, cache files whose file heat value is lower than the second heat threshold are asynchronously deleted.

[0073] Two embodiments are provided below:

[0074] Example 1: Video on demand.

[0075] The client uses an IP connection to upload a data packet containing a video-on-demand request (e.g., the file name of the video being played). The target IP address is a pre-agreed server address. The object storage gateway receives the data packet downloaded by the client, removes the IP header at the application layer, and parses it to obtain the video-on-demand request information. The gateway searches for the corresponding block number based on the file name of the video being played, generates a corresponding hash checksum, and checks the local routing table. If the corresponding data is locally available, the data is directly sent to the client using the IP connection. If the corresponding data is on another storage gateway, an NDN protocol interest packet (containing the required data information) is sent to the other storage gateway, which responds with the corresponding data packet. The storage gateway that receives the data then uses the IP connection to send the on-demand data to the client.

[0076] Example 2: The client reads its own stored data.

[0077] The client calculates the hash value to deduplicate the data in the obtained file list. Users can customize the block size for concurrent downloads, and each file is verified using a hash. If the file block is cached locally, it does not need to be downloaded again. Only the number of downloads of the file block is recorded, reducing the file's consumption of network IO. The storage gateway receives the data packet requested by the client, removes the IP-related header at the application layer, and parses it. It performs relevant proxy forwarding based on the client request. If it is necessary to download a certain data / data shard, the object storage gateway records the data name and shard number, and generates a corresponding hash check code based on the data name and shard number, and searches for it on this gateway or a nearby gateway. The storage gateway client synchronously establishes a routing table, calculates the file heat value based on the file download time, file size, download count, etc. The storage gateway matches the file path with the corresponding hash to establish a heat value routing table.

[0078] A second aspect of the present invention provides: a multi-level cache and file batch downloading system, for implementing any of the above-mentioned multi-level cache and file batch downloading methods, comprising:

[0079] The loading module is used to load the file list using the object storage gateway kernel, and assemble and deduplicate the file list;

[0080] A processing module is used to set a file upload threshold. Files smaller than the file upload threshold are considered small files, while files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented.

[0081] The download module is used to calculate the number of threads required by the cloud storage gateway by adding the number of threads for small file grouping and the number of threads for large file sharding; and to batch start multiple threads for file downloading;

[0082] The cache module is used to cache files according to a first heat threshold after obtaining the files from the cloud storage service end.

[0083] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned multi-level caching and file batch downloading methods is implemented.

[0084] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A multi-level cache and batch file downloading method, characterized by: The following steps are involved: S1: The object storage gateway kernel loads the file list, assembles the file list, and deduplicates it. S2: Set the file upload threshold. Files smaller than the file upload threshold are considered small files, and files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented. S3: Calculate the number of threads required for the cloud storage gateway by adding the number of threads for small file grouping to the number of threads for large file sharding; then enable multiple threads in batches for file downloads. S4: After the cloud storage service obtains the file, it caches it according to the first heat threshold; The S1 further comprises the following steps: The object storage gateway kernel loads the file list and mounts the local Posix disk through the object storage gateway. The object storage gateway schedules with the operating system kernel to obtain the file list, including the file's corresponding hash value, file size, and file attributes, and initiates file download. Assemble and deduplicate the file list, calculate the total number of files, perform integration analysis based on the file hash value, analyze the file duplication, and determine whether there are duplicate files; if there are duplicate file hash values, remove the duplicate files from the download list and record the duplicate file information locally; The S3 further comprises the following steps: After calculating the number of threads required to be opened by the cloud storage gateway, batch-enable multiple threads for file downloading based on the number of threads that can be opened by the storage gateway client. The upload thread batch checks whether there are files with corresponding hash values ​​in the local disk cache based on the file hash values. If the files exist in the local disk cache, the local IO pair is used to open the files directly, and the cloud storage gateway returns the file stream to the application through the system kernel. If the files do not exist in the local disk cache, the storage gateway initiates a batch file download request to the cloud storage gateway server. When the cloud storage service receives a batch download request, it loops through the download file list and compares the hash value of the file in the download file list with the file in the Redis cache. If the file exists, the cloud storage service reads the file stream from the high-speed temporary buffer and outputs it, records the number of file downloads and the download time, and writes it to the Redis cache. If the file does not exist, the cloud storage service initiates a file download request to the object storage gateway. The object storage gateway, based on its own business logic, uses the crash algorithm to obtain the file storage location, accelerates it through the SSD cache, and returns the file stream for the cloud storage service to download. The S4 further comprises the following steps: After the cloud storage service obtains the file, it determines whether it needs to be cached based on the locally recorded file information; if the heat value is higher than the first heat threshold, the file is downloaded to the storage gateway cache area; if the heat value is lower than the first heat threshold, the file is directly input to the storage gateway client through the proxy method; Based on the returned file streams, the storage gateway server combines all file streams into a packaged file stream according to the file size range information; the returned file metadata information records the total data size of the packaged file stream, the size of each file in it, and the corresponding order of the files; The storage gateway client obtains the packaged file stream, parses it, and then downloads it to the local disk. The local disk starts multi-threading based on the number of files and splits the stream file according to the file metadata size, file sequence, and pointer position. Restore each split file according to the corresponding metadata information and dispatch it to the application flow through the kernel; The local disk records the file's corresponding hash value, file download count, and file download time; The local cache detection process calculates the file heat value based on the number of file downloads, file call times and file download time; According to the set local cache space size and the file heat value sorting of each file, cache files whose file heat value is lower than the second heat threshold are asynchronously deleted.

2. The multi-level caching and batch file downloading method according to claim 1, characterized in that: The S2 further comprises the following steps: First, the small files are grouped according to the number of files and the file upload threshold. Then, the file list is looped through and the file sizes are added up to calculate the group size. When the target download file is a large file, the download fragmentation rule is calculated. The number of download fragments = target download file size / file download threshold. The number of groups is increased by 1, and the group size is recalculated. When the target upload file is a large file, the upload fragmentation rule is calculated. The number of upload fragments = the sum of the target upload file sizes / the file upload threshold. When the size of this batch of uploaded files is larger than the file upload threshold, the number of groups is increased by 1. Calculate the overall group size through the loop.

3. A multi-level cache and file batch downloading system, characterized by: The method for implementing the multi-level caching and batch file downloading method according to claim 1 or 2 comprises: The loading module is used to load the file list using the object storage gateway kernel, and assemble and deduplicate the file list; A processing module is used to set a file upload threshold. Files smaller than the file upload threshold are considered small files, while files larger than the file upload threshold are considered large files. Small files are grouped and large files are fragmented. The download module is used to calculate the number of threads required by the cloud storage gateway by adding the number of threads for small file grouping and the number of threads for large file sharding; and to batch start multiple threads for file downloading; The cache module is used to cache files according to a first heat threshold after obtaining the files from the cloud storage service end.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the multi-level cache and file batch downloading method according to claim 1 or 2 is implemented.

Images