A cloud storage file deduplication method based on link technology

Abstract

The present invention discloses a cloud storage file deduplication method based on link technology, which creates and saves link files of duplicate files in cloud storage, and uses link files to save link information. The link information of each link file sequentially records the position of the duplicate file in the cloud storage, realizes the concatenation of the real file and each duplicate file, and the user's access to any duplicate file will find the real file through the traversal of the link file. Since the link file is small and the link information is saved in the cloud storage, it does not rely on local storage, which is conducive to realizing the characteristics of single-point computing and multi-point independent access of file deduplication in a distributed environment, effectively solving the problem of excessive cloud storage space occupation and increased bandwidth cost caused by too many duplicate files, and can be simply and conveniently applied in a distributed system.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud storage and file system, and specifically relates to a cloud storage file deduplication method based on link technology. Background Art

[0002] In the current context, cloud storage technology is often used in multiple scenarios. For example, operators apply for cloud storage services to open up user image and video storage space; users use cloud storage space to share their own storage files for file archiving and data disaster recovery; data service providers use cloud storage for distributed static file caching, etc. These application scenarios have some common characteristics, namely, large storage space occupancy, fast growth in access frequency and bandwidth occupancy, and multiple copies of the same file.

[0003] If you want to store the most files at the lowest cost, it will lead to a comprehensive reduction in storage and transmission costs. Therefore, how to reduce file storage costs has become one of the important considerations for cloud storage solutions.

[0004] There are some technologies on the market that use cloud storage or local storage to perform data deduplication, but most of these technologies rely entirely on local offline calculation results, that is, calculating the hash value of each file, then comparing the hash value, and recording the associated information of these duplicate files locally; when requesting access to the file, use this locally calculated associated information to access and jump. For systems with high distributed requirements, this means that databases need to be deployed at multiple locations and the associated data needs to be synchronized, which is difficult and costly to implement.

[0005] Therefore, a cloud storage file deduplication method with better performance is needed. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a cloud storage file deduplication method with better performance.

[0007] In order to solve the above technical problems, the present invention provides a cloud storage file deduplication method based on link technology, which creates and saves link files of duplicate files in cloud storage, and uses link files to save link information. The link information of each link file sequentially records the location of the duplicate file in the cloud storage, and realizes the concatenation of the real file and each duplicate file. When a user accesses any duplicate file, the real file will be found by traversing the link file. Since the link file is small and the link information is saved in the cloud storage and does not rely on local storage, it is conducive to realizing the characteristics of single-point computing and multi-point independent access of file deduplication in a distributed environment, effectively solving the problem of excessive cloud storage space occupation and increased bandwidth cost caused by too many duplicate files, and at the same time can be simply and conveniently applied in a distributed system; it is characterized by comprising the following steps:

[0008] Step 1: Configure a local file information table, which is used to record file information of existing files in the cloud storage system. The file information includes file content and storage path. The configuration items of the file information table include a hash value of the file content and a storage path of the file.

[0009] Step 2: Obtain the file content of the file to be uploaded, parse the file content, and calculate the hash value of the file content; obtain the local file information table, and search the file information table for file information that is consistent with the hash value of the file content; if not, determine that the file to be uploaded is a real file, and execute step 3; if so, determine that the file to be uploaded is a duplicate file, and execute step 4;

[0010] Step 3: Call the file upload interface of the cloud storage system, upload the file information of the real file to the cloud storage system, and record the hash value of the file content and the file storage path in the file information table;

[0011] Step 4: Create a new link file for the duplicate file to be uploaded this time, store the file information of the link file in the cloud storage system, and record the hash value of the file content of the link file and the file storage path in the file information table, where the hash value of the file content of the link file is the hash value of the file content calculated in step 2;

[0012] Step 5: traverse the file information of existing files in the cloud storage system, perform file deduplication access, and obtain the file information of real files in the cloud storage system.

[0013] The algorithm for calculating the hash value in step 2 is MD5; the cloud storage system in step 2 is Amazon S3.

[0014] The step 4 includes: marking a newly created link file with a preset text key-value pair; writing a magic value, a version number and a link position of the link file in the file content of the newly created link file; the magic value and the version number of the link file are both preset identifiers with a fixed number of bytes; obtaining file information of an existing file having the same hash value as the file content in step 2 from a file information table, and taking the storage path of the last uploaded existing file as the link position in the order in which the existing files are uploaded; and storing the file information of the file to be linked in a cloud storage system, saving the text key-value pair in the form of file metadata.

[0015] In step 4, the magic value, version number and link position of the link file are written into the file content of the newly created link file, and the magic value, version number and link position of the link file are written in sequence starting from the head of the file content;

[0016] Step 5 includes: extracting the storage path in the file information access request, and obtaining the file content and text key-value pairs from the cloud storage system according to the storage path; comparing whether the text key-value pairs are consistent with the text key-value pairs of the linked file in Step 4; if they are inconsistent, determining that the currently obtained file is a real file; returning the file information of the real file to the browser side; if they are consistent, determining that the currently obtained file is a linked file, extracting the link position in the file content, and continuing to obtain the file content and text key-value pairs from the cloud storage system according to the storage path pointed to by the link position, and comparing the text key-value pairs until it is determined that the file is a real file according to the comparison result.

[0017] In Step 4, the key of the text key-value pair is Content-Type, and the value is a custom MIME type; the text key-value pair of the linked file is expressed as <Content-Type, custom MIME type>; the storage of the text key-value pair is implemented through the file metadata function of Amazon S3; the text key-value pair in the file metadata is obtained while obtaining the file information from Amazon S3.

[0018] In Step 5, the comparison of the text key values specifically includes:

[0019] Step 501: Comparing whether the text key-value pair is a key-value pair with the key of Content-Type, and whether the value in the key-value pair with the key of Content-Type is consistent with the custom MIME type in Step 4; if both of the above are satisfied, the file is determined to be a linked file; if both of the above are not satisfied, the file is determined to be a real file;

[0020] Step 502: If the file is determined to be a linked file, calculate the fixed number of bytes occupied by the linked file magic value and the linked file version number from the head of the linked file content, and use the position of the fixed number of bytes plus 1 as the starting position, and extract all the file content from the starting position as the link position.

[0021] In Steps 1-5, if the cloud storage system does not support the storage of MIME types, it is determined whether the file is a linked file according to the linked file magic value. Specifically, according to the number of bytes occupied by the preset linked file magic value, the same number of bytes of content is extracted from the head of the file content. If the extracted content is consistent with the linked file magic value pre-configured in Step 4, it is determined to be a linked file; if not, it is determined to be a real file.

[0022] The beneficial effects achieved by the present invention are:

[0023] The present invention calculates the hash value of the cloud storage file offline, creates a link file with extremely small content for the duplicate file, reduces the capacity of the cloud storage, and reduces the bandwidth occupied when uploading files. Compared with the prior art, the link relationship storage required for distributed system access is reduced, and the cost of local storage is reduced. In addition, although multiple accesses seem to reduce the performance of file access, the access speed is extremely fast because the link file is extremely small, and the performance loss is almost negligible. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the process of implementing the file deduplication method based on link technology in an embodiment of the present invention

[0025] Figure 2 A schematic diagram of the structure of the link file content in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a process for deduplicating file access in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the system structure for deduplication of files in cloud storage based on linking technology in an embodiment of the present invention;

[0028] Figure 5 A system deployment diagram for deduplication of cloud storage files based on linking technology in an embodiment of the present invention; DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and exemplary embodiments.

[0030] In the embodiment of the present invention, Amazon S3 is used as a cloud storage system; the Amazon S3, full name of which is Amazon Simple Storage Service, is a service provided by Amazon Web Services (AWS), which provides object storage services through a variety of communication methods. Amazon S3 can be used to store any type of objects, which can be used for storage, backup and recovery, disaster recovery, data archiving, data lakes for analysis, and hybrid cloud storage similar to Internet applications. For example, when a user uploads a picture to the Amazon S3 cloud storage system, the Amazon S3 cloud storage system synchronizes the pictures uploaded to the cloud storage system to the various edge nodes of Amazon S3 to help users quickly display pictures to target customers in various places around the world, so as to speed up local customer access.

[0031] like Figure 1 The schematic diagram of the implementation flow of a file deduplication method based on link technology in an embodiment of the present invention shown in the figure specifically includes:

[0032] Step 101: The storage path configures a local file information table, which is used to record the file information of existing files in the Amazon S3 cloud storage system. The file information includes file content and storage path. The configuration items of the file information table include the hash value of the file content and the storage path of the file. After any file is uploaded to the Amazon S3 cloud storage system, the storage path is obtained and stored in the file information table. The files include documents, pictures, videos and audios.

[0033] Step 102: Calculate the hash value to determine whether the file to be uploaded is a duplicate file, specifically: obtain the file information to be uploaded to the Amazon S3 cloud storage system, parse the file content, and use the MD5 algorithm to calculate the hash value of the file content; search in the file information table whether there is file information consistent with the hash value, if so, there is file information with the same content as the file in the cloud storage system, and the file to be uploaded is determined to be a duplicate file, and step 103 is executed; if not, there is no file information with the same content as the file in the cloud storage system, and the file to be uploaded is determined to be a real file, and the upload service interface is called to upload the file to the cloud storage system, and the file content hash value and storage path of the file are saved in the file information table;

[0034] Step 103: Create a new link file for the duplicate file to be uploaded this time, store the file information of the link file in the cloud storage system, and record the hash value of the file content of the link file and the file storage path in the file information table, where the hash value of the file content of the link file is the hash value of the file content calculated in step 102;

[0035] The specific process includes: first, marking the newly created link file with a preset text key-value pair; second, writing the magic value, version number and link position of the link file in the file content of the newly created link file; the magic value and version number of the link file are both preset identifiers with a fixed number of bytes; obtaining the file information of the existing file that is the same as the hash value of the file content in step 102 from the file information table, and taking the storage path of the last uploaded existing file as the link position according to the time sequence of the existing file upload; finally, storing the file information of the file to be linked in the cloud storage system, and saving the text key-value pair in the form of file metadata;

[0036] The specific process of creating a link file includes:

[0037] Step 103-1: Write the content of the link file, including writing the magic value of the file type, the link file version number, and the link location in sequence;

[0038] Writing the magic value of the link file: writing a preset magic value of the link file into the file header, such as writing "link";

[0039] Writing the link file version number: after the link file magic value, write the preset link file version number into the file content, for example, write "V1.0" after "link";

[0040] Writing the link position: obtaining file information records with the same hash value as the duplicate file from the file information table, sorting them from front to back according to the time when the files were uploaded, taking the cloud storage position in the last file information record as the value of the link position, and writing it after the link file version number;

[0041] The write code is configured in the file upload program, as follows:

[0042] private byte[]createLinkFile(String fileLocation,byte version){

[0043] / / Define the magic value

[0044] final String MAGIC_NUMBER="link";

[0045] / / Concatenate three groups of bytes in order, namely the magic number bytes, the version bytes, and the file location

[0046] return combineBytes(MAGIC_NUMBER.getBytes(StandardCharsets.UTF_8),newbyte[]{version},fileLocation.getBytes(StandardCharsets.UTF_8));

[0047] like Figure 2 The structure diagram of the link file content in the exemplary embodiment of the present invention is shown in the figure. The link file content starts with the link file magic value "LINK", followed by the version number "V0.1", and all bytes after the version number are the position of the previous link file of the duplicate file in the cloud storage, such as "POSITION" in the figure. Since the link file magic value and the link file version number are preset values ​​and have a fixed number of bytes; therefore, starting from the file header, all bytes after the fixed number of bytes are the link position;

[0048] Step 103-2: Marking the newly created link file with a preset text key-value pair, specifically: in the process of uploading the file to the Amazon S3 cloud storage system, the file upload program adds a preset text key-value pair to the file; in the text key-value pair, the key is Content-Type, and the value is the MIME type; the MIME type includes a conventional MIME type value and a custom MIME type value; using the file metadata function of the Amazon S3 cloud storage system, the text key-value pair is saved in the file metadata of the file uploaded to Amazon S3;

[0049] For non-link files, the MIME type is the conventional MIME type value disclosed according to the file type tag; for example, image / jpg, text / html, etc. For link files, the MIME type is a custom MIME type, such as "application / vnd.mic-link";

[0050] The code for marking the link file is configured in the file upload program as follows:

[0051]

[0052]

[0053] In the above code, the text key-value pair is written into the file upload program. When uploading a file, a key-value pair is added to the file. Taking the link file as an example, write<Content-Type,application / vnd.mic-link> ; The key-value pair will be used to determine whether the file is a link file when obtaining the file;

[0054] The structure of MIME type (Multipurpose Internet Mail Extensions, a public definition of Internet file transfer types) is type / subtype; it consists of two strings, type and subtype, separated by ' / '. No spaces are allowed. Type represents an independent category that can be divided into multiple subcategories; subtype represents each type after subdivision. Everyone can customize MIME types. In order to avoid duplication and misunderstanding, individuals or companies need to add the prefix vnd to the subtype when formulating agreements, supplemented by the entity identifier, and then the file type. For example, application / vnd.ms-excel, where vnd is the identifier that this MIME type is customized, ms is Microsoft, and excel is the actual format of the file.

[0055] Step 104: traverse the file information of existing files in the cloud storage system and perform file de-revisit.

[0056] like Figure 3 The flowchart of file deduplication access in the exemplary embodiment of the present invention shown in FIG. 1 includes:

[0057] Step 104-1: The user requests to access the file information of an existing file in the Amazon S3 cloud storage system through a browser, and the server of the cloud storage system responds to the request, obtains the storage path in the file information request, and extracts the file information and file metadata from the Amazon S3 cloud storage system according to the storage path;

[0058] Step 104-2: extracting a text key-value pair with a key of Content-Type from the file metadata, and comparing whether the value in the text key-value pair is a conventional MIME type value or a custom MIME type value, wherein the custom MIME type value is application / vnd.mic-link in step 103;

[0059] If the value in the text key-value pair is a conventional value of the MIME type, the currently acquired file is determined to be a real file; and the file information of the real file is returned to the browser end;

[0060] No, if the value in the text key-value pair is application / vnd.mic-link, the obtained file is determined to be a link file, the file information of the link file is extracted, the link position in the file content is parsed, and the storage path of the file information of the previous duplicate file in the Amazon S3 cloud storage system is obtained;

[0061] 104-3: according to the storage path of step 104-2, extract the file information from the Amazon S3 cloud storage system, and determine the file according to step 104-2 until the file is determined to be a real file and return the file content to the browser;

[0062] In the embodiment of the present invention, since the preset link file magic value and link file version number occupy a fixed number of bytes, for example, the agreed link file magic value is "MIC-LINK", which occupies 8 bytes, and then the link file version number 1.0 occupies one byte, a total of 9 bytes, then from the 10th byte onwards are link positions.

[0063] In the embodiment of the present invention, when the access workstations deployed in various places access the Amazon S3 cloud storage system site at the same time, each workstation performs MIME type judgment on the extracted file to confirm whether the file is a link file. Since the Amazon S3 service itself can mark the file type for it, a custom MIME type is set for the link file separately: application / vnd.xx-link. In this way, when accessing pictures, the type of the picture can be quickly known. For ordinary pictures, we use its original MIME format, such as image / png, image / jpg, etc.

[0064] like Figure 4 The schematic diagram of the system structure of deduplication of files in cloud storage based on link technology in the exemplary embodiment of the present invention shown in the figure specifically includes a local file server, an application server, a cloud storage system and a file information data table configured in a local computer room; the file deduplication system of the present invention is deployed in the application server;

[0065] The local file server is used to initiate an upload request for file information;

[0066] The application server is equipped with a duplicate calculation module, a file deduplication module, a file upload module, and a file access module;

[0067] The duplicate calculation module is used to perform hash value calculation on the file content of the uploaded file and call the file information data table to perform hash value comparison, thereby determining whether the file to be uploaded is duplicated with the file in the cloud storage;

[0068] The file deduplication is to perform different operations according to whether the file is repeated or not, and the non-duplicate files are handed over to the file upload module; the duplicate files are generated into a link file and handed over to the file upload module;

[0069] The file upload module is used to call the file upload interface of the cloud storage system to upload the real file and the link file to the cloud storage system;

[0070] The file access module is used to obtain files in the cloud storage system according to the file storage location;

[0071] The cloud storage system is an Amazon S3 cloud storage system, which is used to store real files and link files corresponding to duplicate files;

[0072] Deploy at least one upload server in the local computer room; allow multiple upload servers to perform file uploads concurrently;

[0073] The storage location of the link file of the previous duplicate file in the cloud storage system is saved in the link file content. By saving the link information of the storage location of the real file and multiple duplicate files on the cloud server, file access deduplication no longer relies on the local file information data table, eliminating the associated data synchronization step in the distributed environment, and achieving access deduplication within the cloud storage system in a simpler and lower-cost manner.

[0074] like Figure 5 The system deployment diagram of a cloud storage file deduplication based on link technology in an exemplary embodiment of the present invention shown includes a file upload server, a file record table, a cloud storage system, and an access agent; at the system deployment level. Multiple upload servers can concurrently execute the task of uploading image files. They all rely on a local file record table. Relying on this local file record table, the upload server can compare MD5 information to determine whether a specific image has been uploaded before. The access agent is the access service of the cloud storage, which does not rely on the local file record table - because the link information has been uploaded to the cloud in the form of a link file. Multiple access agents can operate concurrently at the same time.

[0075] The present invention provides a cloud storage file deduplication method based on link technology, which has the following beneficial effects:

[0076] By calculating the hash value of the cloud storage file offline, a link file with extremely small content is created for the duplicate file, which reduces the capacity of cloud storage and reduces the bandwidth occupied when uploading files. Compared with the existing technology, the link relationship storage required for distributed system access is reduced, and the cost of local storage is reduced. In addition, although multiple accesses seem to reduce the performance of file access, the access speed is extremely fast due to the extremely small link file, and the performance loss is almost negligible.

[0077] The above embodiments do not limit the present invention in any way. Any other improvements and applications made to the above embodiments in an equivalent transformation manner belong to the protection scope of the present invention.

Claims

1. A cloud storage file deduplication method based on link technology, characterized in that: The method comprises the following steps: Step 1: configuring a local file information table, wherein the file information table is used to record file information of existing files in the cloud storage system, wherein the file information includes file content and storage path; the configuration items of the file information table include a hash value of the file content and a storage path of the file; Step 2: Get the file content of the file to be uploaded, parse the file content, and calculate the hash value of the file content; Obtain a local file information table, and search the file information table for file information that is consistent with the hash value of the file content; If it does not exist, determine that the file to be uploaded is a real file and execute step 3; If so, determine that the file to be uploaded is a duplicate file, and execute step 4; Step 3: Call the file upload interface of the cloud storage system, upload the file information of the real file to the cloud storage system, and record the hash value of the file content and the file storage path in the file information table; Step 4: Create a new link file for the duplicate file to be uploaded this time, store the file information of the link file in the cloud storage system, and record the hash value of the file content of the link file and the file storage path in the file information table, where the hash value of the file content of the link file is the hash value of the file content calculated in step 2; The step 4 includes: marking the newly created link file with a preset text key-value pair; writing the magic value, version number and link position of the link file in the file content of the newly created link file; the magic value and version number of the link file are both preset identifiers with a fixed number of bytes; obtaining the file information of the existing file having the same hash value as the file content in step 2 from the file information table, and taking the storage path of the last uploaded existing file as the link position according to the time sequence of the existing file uploads; storing the file information of the file to be linked in the cloud storage system, and saving the text key-value pair in the form of file metadata; Step 5: traverse the file information of existing files in the cloud storage system, perform file deduplication access, and obtain the file information of real files in the cloud storage system; The step 5 includes: extracting the storage path in the file information access request, obtaining the file content and text key-value pairs from the cloud storage system according to the storage path; comparing whether the text key-value pairs are consistent with the text key-value pairs of the link file in step 4; if they are inconsistent, determining that the currently obtained file is a real file; returning the file information of the real file to the browser; if they are consistent, determining that the currently obtained file is a link file, extracting the link position in the file content, continuing to obtain the file content and text key-value pairs from the cloud storage system according to the storage path pointed to by the link position, and comparing the text key-value pairs until the file is determined to be a real file according to the comparison result.

2. A cloud storage file deduplication method based on link technology as claimed in claim 1, characterized in that: The algorithm for calculating the hash value in step 2 is MD5; the cloud storage system in step 2 is Amazon S3.

3. A cloud storage file deduplication method based on link technology as claimed in claim 1, characterized in that: In step 4, the magic value, version number and link position of the link file are written into the file content of the newly created link file, and the magic value, version number and link position of the link file are written in sequence starting from the head of the file content.

4. A cloud storage file deduplication method based on linking technology as claimed in claim 1, characterized in that: In step 4, the key of the text key-value pair is Content-Type, and the value is a custom MIME type; the text key-value pair of the linked file is represented as <Content-Type, custom MIME type>; the storage of the text key-value pair is implemented through the file metadata function of Amazon S3; when obtaining file information from Amazon S3, the text key-value pair in the file metadata is obtained simultaneously.

5. A cloud storage file deduplication method based on link technology as claimed in claim 1, characterized in that: In step 5, the comparison of text keys specifically includes: Step 501: Check whether the text key-value pair is a key-value pair with the key Content-Type, and whether the value in the key-value pair with the key Content-Type is consistent with the custom MIME type in step 4; if both of the above are satisfied, the file is determined to be a linked file; if both of the above are not satisfied, the file is determined to be a real file; Step 502: If the file is determined to be a linked file, calculate the fixed number of bytes occupied by the magic value and version number of the linked file starting from the head of the linked file content, and use the position one more than the fixed number of bytes as the starting position, and extract all the file content starting from the starting position as the link position.

6. A cloud storage file deduplication method based on linking technology as claimed in claim 1, characterized in that: In steps 1-5, if the cloud storage system does not support the storage of MIME types, the file is determined whether it is a linked file according to the magic value of the linked file. Specifically, according to the number of bytes occupied by the preset magic value of the linked file, the same number of bytes of content is extracted from the head of the file content. If the extracted content is consistent with the magic value of the linked file pre-configured in step 4, it is determined to be a linked file; if not, it is determined to be a real file.

Images