A cloud storage deduplication method and device based on sliding window block optimization algorithm

Abstract

The present invention belongs to the technical field of deduplication of cloud storage, and specifically relates to a cloud storage deduplication method and device based on a sliding window block optimization algorithm. The cloud storage deduplication method is based on a sliding window block optimization algorithm, and determines the current position of the sliding window by moving the sliding window, and compares it with the file position to perform different deduplication steps. The main idea of ​​deduplication is to perform deduplication by multiple block divisions and calculating hash values, and judging whether there are repeated hash values. It can more effectively increase the deduplication rate of the block-level deduplication method and improve the network bandwidth utilization. It has the advantages of high network bandwidth utilization and easy implementation, and fully utilizes the characteristics of Bloom filters to reduce the memory occupied by files, and uses extremely small space to gain great performance advantages.

Description

Technical Field

[0001] The present invention belongs to the technical field of deduplication of cloud storage, and in particular relates to a cloud storage deduplication method and device based on a sliding window block optimization algorithm. Background Art

[0002] With the advent of the big data era, enterprises and individuals continue to generate massive amounts of data, and important data needs to be stored securely. Cloud storage can integrate all storage resources together and improve the utilization of storage space through virtualization technology. However, during the transmission and storage process, the redundant data in the middle will waste a lot of storage space and network bandwidth. Cloud storage deduplication technology saves a lot of storage space for users, greatly improves the utilization of network bandwidth, and is increasingly used in different fields of real life, greatly facilitating people's production and life.

[0003] Cloud storage deduplication technology can be divided into file-level and block-level deduplication according to the deduplication granularity. Different deduplication granularities have different focuses. File-level deduplication has a low deduplication rate, and even the smallest changes cannot be recognized. Block-level deduplication has a higher deduplication rate, but fixed-block deduplication also has high requirements for files. If a part of the data is added, it is very likely that it will not be recognized at all.

[0004] Therefore, a new deduplication method is urgently needed to solve the problem of low deduplication rate of block-level deduplication methods. Summary of the invention

[0005] In view of the above problems existing in the prior art, the present invention provides a cloud storage deduplication method and device based on a sliding window block optimization algorithm, which can more effectively increase the deduplication rate and improve the network bandwidth utilization.

[0006] The present invention adopts the following technical solutions:

[0007] A cloud storage deduplication method based on a sliding window block optimization algorithm comprises the following steps:

[0008] S1, upload the file, determine the first data position of the sliding window as start, the last data position of the sliding window as end, and the starting position of the file as mark;

[0009] S2, if end ≤ the end of the file, execute step S3, if end > the end of the file and start ≤ the end of the file, execute step S4, if start > the end of the file, execute step S8;

[0010] S3, calculate the hash value of the data in the sliding window, and determine whether the hash value exists through the filter. If not, move the sliding window downward by a first preset distance, update start and end, and execute step S2. If it exists, execute step S5;

[0011] S4, calculate the hash value of the data in the sliding window, and determine whether the hash value exists in the database. If not, move the sliding window downward by a first preset distance, update start and end, and execute step S2. If it exists, execute step S5;

[0012] S5, determine whether the current start and mark are at the same position, if so, move the sliding window downward by a second preset distance, update start and end, and execute step S2, if not, execute step S6;

[0013] S6, divide the corresponding content into blocks, calculate the hash value of each block, determine whether the hash value exists through the filter, if it exists, do not upload to the server, if not, map its value into the filter, and divide the non-existent block into multiple sub-blocks again, calculate the hash value of each sub-block, determine whether the hash value exists in the database, if not, upload it to the server and record it in the database, if it exists, do not upload to the server, and execute step S7;

[0014] S7, the sliding window moves downward as a whole by a second preset distance, updates start and end, assigns the value of start to mark, and returns to step S2;

[0015] S8. Determine whether the start and mark are at the same position. If so, end the upload. If not, divide the corresponding content into blocks, calculate the hash value of each block, and determine whether the hash value exists through the filter. If it exists, do not upload it to the server. If it does not exist, map its value into the filter, and divide the non-existent block into multiple sub-blocks again. Calculate the hash value of each sub-block and determine whether the hash value exists in the database. If it does not exist, upload it to the server and record it in the database. If it exists, do not upload it to the server.

[0016] As a preferred solution, in step S3 and step S4, the first preset distance is specifically a byte distance;

[0017] In step S5 and step S7, the second preset distance is specifically a distance of a sliding window.

[0018] As a preferred solution, in step S6, the content from mark to the byte before start is divided into blocks.

[0019] As a preferred solution, in step S6, the content from the mark to the byte before the start is divided into blocks according to the length of the sliding window.

[0020] As a preferred solution, in step S8, the content marked to the end of the file is divided into blocks.

[0021] As a preferred solution, in step S8, the content marked to the end of the file is divided into blocks according to the length of the sliding window.

[0022] As a preferred solution, in step S6 and step S8, after the corresponding content is divided into blocks according to the length of the sliding window, if there are remaining blocks whose length is less than the sliding window, the hash value of the remaining block is calculated to determine whether the hash value exists in the database. If not, it is uploaded to the server and recorded in the database. If it exists, it is not uploaded to the server.

[0023] As a preferred solution, the filter adopts Bloom filter.

[0024] As a preferred solution, in both step S6 and step S8, the block is divided into three sub-blocks.

[0025] Accordingly, a cloud storage deduplication device based on a sliding window block optimization algorithm is also provided, comprising:

[0026] A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the above method.

[0027] The beneficial effects of the present invention are as follows: based on the sliding window block optimization algorithm, the current position of the sliding window is determined by moving the sliding window, and compared with the file position to perform different deduplication steps. The main idea of ​​deduplication is to perform deduplication by multiple block divisions and calculating hash values, and judging whether there are repeated hash values. It can more effectively increase the deduplication rate of the block-level deduplication method and improve the network bandwidth utilization. It has the advantages of high network bandwidth utilization and easy implementation. It makes full use of the characteristics of the Bloom filter, reduces the memory occupied by the file, and uses a very small space to gain a great performance advantage. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1It is a flow chart of a first route of a cloud storage deduplication method based on a sliding window block optimization algorithm described in the present invention;

[0030] Figure 2 It is a flow chart of a second route of a cloud storage deduplication method based on a sliding window block optimization algorithm described in the present invention;

[0031] Figure 3 It is a flowchart of the third route of the cloud storage deduplication method based on the sliding window block optimization algorithm described in the present invention. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0033] Embodiment 1:

[0034] Reference Figure 1 , 2 As shown in FIG. 3 , this embodiment provides a cloud storage deduplication method based on a sliding window block optimization algorithm, comprising the steps of:

[0035] S1, upload the file, determine the first data position of the sliding window as start, the last data position of the sliding window as end, and the starting position of the file as mark;

[0036] S2, if end≤the end position of the file (the end position of the file is the position of the last byte of the file), execute step S3 (for specific steps, please refer to Figure 1 As shown), if end>the end position of the file and start≤the end position of the file, execute step S4 (the specific step flow can be referred to Figure 2 As shown), if start> the end of the file, execute step S8 (for specific steps, refer to Figure 3 shown);

[0037] S3, calculate the hash value of the data in the sliding window, and determine whether the hash value exists through the filter. If not, move the sliding window downward by a first preset distance (the first preset distance in this embodiment is the first byte distance), update start and end, and execute step S2. If it exists, execute step S5;

[0038] S4, calculate the hash value of the data in the sliding window, and determine whether the hash value exists in the database. If not, move the sliding window downward by a first preset distance, update start and end, and execute step S2. If it exists, execute step S5;

[0039] S5, determine whether the current start and mark are at the same position, if so, move the entire sliding window downward by a second preset distance (in this embodiment, the second preset distance is the distance of a sliding window), update start and end, and execute step S2, if not, execute step S6;

[0040] S6, divide the corresponding content into blocks, specifically divide the content from the mark to the byte before the start into blocks according to the length of the sliding window, calculate the hash value of each block, determine whether the hash value exists through the filter, if it exists, do not upload to the server, if not, map its value into the filter, and divide the non-existent block into multiple sub-blocks again, calculate the hash value of each sub-block, determine whether the hash value exists in the database, if not, upload it to the server and record it in the database, if it exists, do not upload to the server, and execute step S7;

[0041] S7, the sliding window moves downward as a whole by a second preset distance, updates start and end, assigns the value of start to mark, and returns to step S2;

[0042] S8. Determine whether start and mark are at the same position. If so, end the upload. If not, divide the corresponding content into blocks. Specifically, divide the content from mark to the end of the file into blocks according to the length of the sliding window, calculate the hash value of each block, and use the filter to determine whether the hash value exists. If so, do not upload it to the server. If not, map its value into the filter, and divide the non-existent block into multiple sub-blocks again. Calculate the hash value of each sub-block, determine whether the hash value exists in the database, if not, upload it to the server and record it in the database, if it exists, do not upload it to the server.

[0043] It should be noted that the attached Figure 1 , 2 , 3 are different routes in the same method, but they are divided into three pictures for display.

[0044] Specifically:

[0045] In step S6 and step S8, after the corresponding content is divided into blocks according to the length of the sliding window, if there are residual blocks whose length is less than the sliding window, the hash value of the residual block is calculated to determine whether the hash value exists in the database. If not, it is uploaded to the server and recorded in the database. If it exists, it is not uploaded to the server.

[0046] In this embodiment, in both step S6 and step S8, the block is divided into three sub-blocks.

[0047] In this embodiment, the filter adopts Bloom filter. Bloom filter: Bloom filter is a binary vector data structure proposed by Howard Bloom in 1970. It has space and time efficiency and is used to detect whether an element is a member of a set.

[0048] The cloud storage deduplication method provided in this embodiment is based on a sliding window block optimization algorithm. By moving the sliding window, the current position of the sliding window is determined and compared with the file position to perform different deduplication steps. The main idea of ​​deduplication is to perform deduplication by dividing the blocks multiple times and calculating the hash values, and judging whether there are repeated hash values. It can more effectively increase the deduplication rate of the block-level deduplication method and improve the network bandwidth utilization. It has the advantages of high network bandwidth utilization and easy implementation. It makes full use of the characteristics of the Bloom filter, reduces the memory occupied by the file, and uses a very small space to gain a huge performance advantage.

[0049] Embodiment 2:

[0050] This embodiment provides a cloud storage deduplication device based on a sliding window block optimization algorithm, including:

[0051] A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method described in the first embodiment.

[0052] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0053] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0054] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0055] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A cloud storage deduplication method based on a sliding window block optimization algorithm, characterized in that: Includes steps: S1, upload the file, determine the first data position of the sliding window as start, the last data position of the sliding window as end, and the starting position of the file as mark; S2, if end ≤ the end of the file, execute step S3, if end > the end of the file and start ≤ the end of the file, execute step S4, if start > the end of the file, execute step S8; S3, calculate the hash value of the data in the sliding window, and determine whether the hash value exists through the filter. If not, move the sliding window downward by a first preset distance, update start and end, and execute step S2. If it exists, execute step S5; S4, calculate the hash value of the data in the sliding window, and determine whether the hash value exists in the database. If not, move the sliding window downward by a first preset distance, update start and end, and execute step S2. If it exists, execute step S5; S5, determine whether the current start and mark are at the same position, if so, move the sliding window downward by a second preset distance, update start and end, and execute step S2, if not, execute step S6; S6, divide the corresponding content into blocks, calculate the hash value of each block, determine whether the hash value exists through the filter, if it exists, do not upload to the server, if not, map its value into the filter, and divide the non-existent block into multiple sub-blocks again, calculate the hash value of each sub-block, determine whether the hash value exists in the database, if not, upload it to the server and record it in the database, if it exists, do not upload to the server, and execute step S7; S7, the sliding window moves downward as a whole by a second preset distance, updates start and end, assigns the value of start to mark, and returns to step S2; S8, determine whether the start and mark are at the same position, if so, end the upload, if not, divide the corresponding content into blocks, calculate the hash value of each block, determine whether the hash value exists through the filter, if it exists, do not upload to the server, if not, map its value into the filter, and divide the non-existent block into multiple sub-blocks again, calculate the hash value of each sub-block, determine whether the hash value exists in the database, if not, upload it to the server and record it in the database, if it exists, do not upload to the server; In step S3 and step S4, the first preset distance is specifically a byte distance; In step S5 and step S7, the second preset distance is specifically a distance of a sliding window.

2. According to claim 1, a cloud storage deduplication method based on a sliding window block optimization algorithm is characterized in that: In step S6, specifically, the content from mark to the byte before start is divided into blocks.

3. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 2 is characterized in that: In step S6, specifically, the content from the mark to the byte before the start is divided into blocks according to the length of the sliding window.

4. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 3 is characterized in that: In step S8, specifically, the content marked to the end of the file is divided into blocks.

5. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 4 is characterized in that: In step S8, specifically, the content marked to the end of the file is divided into blocks according to the length of the sliding window.

6. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 5 is characterized in that: In step S6 and step S8, after the corresponding content is divided into blocks according to the length of the sliding window, if there are residual blocks whose length is less than the sliding window, the hash value of the residual block is calculated to determine whether the hash value exists in the database. If not, it is uploaded to the server and recorded in the database. If it exists, it is not uploaded to the server.

7. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 1 is characterized in that: The filter uses Bloom filter.

8. The cloud storage deduplication method based on sliding window block optimization algorithm according to claim 1 is characterized in that: In both step S6 and step S8, the block is divided into three sub-blocks.

9. A cloud storage deduplication device based on a sliding window block optimization algorithm, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images