Data deduplication system and method based on inter-network security IP tunnel

Abstract

The invention provides a data deduplication system and method based on an inter-network security IP tunnel, and the system comprises the inter-network security IP tunnel which is established between a local area network and a data center based on a programmable switch. The encryption module is used for encrypting the whole data packet related to data deduplication leaving the local area network and the data center and decrypting the whole data packet related to data deduplication entering the local area network and the data center; the edge switch corresponding to the storage server is used for completing data deduplication and encrypted online calculation; and the centralized controller in the data center network is responsible for carrying out key negotiation with the controller in the local area network. According to the method, finer duplicate removal granularity can be supported, so that the efficiency of deleting the duplicate data can be remarkably improved while acceptable extra storage cost is introduced and CPU (Central Processing Unit) consumption of a server is reduced.

Description

Technical Field

[0001] The invention belongs to the field of network systems, and in particular relates to a data deduplication system and method based on an inter-network secure IP tunnel. Background Art

[0002] With the continuous development of Internet technology and the continuous growth of the total number of Internet users, the total amount of data in the world has also increased significantly. In this case, how to manage data storage economically and efficiently has become one of the most challenging and important tasks in the massive storage system in the big data era. At the same time, due to the storage capacity of a single device, individuals and organizations often need to rely on cloud storage service providers to achieve low-cost storage, transmission and backup of growing data. In this case, in order to further improve the efficiency of data storage and reduce the cost of data storage, common cloud service providers usually use data deduplication technology to avoid multiple storage of duplicate data as much as possible, thereby reducing the overhead of data storage and reducing the user's upload bandwidth.

[0003] At the same time, considering that there are a large number of malicious attackers in the real network environment, in order to avoid the leakage of user privacy data, it is often necessary to encrypt the real plaintext data before transmission. However, this will result in different users uploading the same data, each using different private keys for encryption, resulting in the same plaintext eventually becoming different ciphertexts, so that the data storage server cannot identify data duplication.

[0004] In current network research, there are two traditional solutions, namely, the method based on Convergent Encryption and the method based on SSL or TLS. For the data plaintext M, the method based on Convergent Encryption will first use a hash function to calculate the hash value H(M) of M, and use this hash value as the encryption key to encrypt the plaintext to obtain E(H(M), M). At the same time, in order to ensure that users can still decrypt the data when downloading files, the data center needs to store the hash value H(M) as the encryption key. For security reasons, the hash value H(M) needs to be encrypted by the user's respective private key to obtain E(Ka, H(M)). The method based on SSL or TLS directly encrypts the data using the SSL or TLS protocol during transmission, and then decrypts it using the SSL or TLS protocol when the data storage server receives it before subsequent operations can be performed. For the convergent encryption method, because the encryption result of the hash key needs to be stored additionally, the block size for data deduplication can basically only be file level or relatively large block (KB level). Otherwise, when the block size is very small, the additional storage overhead cannot be ignored, so it is impossible to fully utilize the redundant information in the file for deduplication, and the deduplication effect is poor. For the SSL or TLS method, since the storage data server uses plain text when determining data duplication, if the user's private data is directly stored in plain text, there will be a greater security risk. Therefore, the data still needs to be encrypted before each actual data storage, and the data needs to be decrypted each time the data is downloaded, which increases the CPU overhead of the server.

[0005] Therefore, it is an urgent problem to be solved by technical personnel in this field to provide a system / method that can perform deduplication with a smaller block size and offload complex encryption operations on the server side to online computing. This system / method can greatly increase the deduplication efficiency of the data storage system and significantly reduce the CPU overhead on the server side. Summary of the invention

[0006] In view of the defect that the existing storage server file transmission data deduplication cannot take into account both the data deduplication storage efficiency and the storage server-side CPU overhead at the same time, the present invention provides a data deduplication system and method based on an inter-network secure IP tunnel, which can ensure the security and reliability of data packet transmission between a local area network and a data center network by establishing an encrypted inter-network IP tunnel, thereby ensuring the privacy of user privacy data in an untrusted transmission environment with malicious nodes while enabling the plaintext of user data to be obtained inside the data center network, thereby facilitating subsequent data analysis and processing without introducing additional storage overhead, and offloading the encryption and decryption operations of the stored data to the edge programmable switch, thereby achieving a significant improvement in deduplication efficiency and an improvement in server-side CPU utilization, saving the overhead and cost of the data center storage server.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a data deduplication system based on an internetwork security IP tunnel, comprising: an internetwork security IP tunnel, an edge switch and a centralized controller;

[0009] The inter-network secure IP tunnel is established based on a programmable switch, which is located between the local area network and the data center, and is used to encrypt the entire packet of data packets related to data deduplication leaving the local area network and the data center, and to decrypt the entire packet of data packets related to data deduplication entering the local area network and the data center;

[0010] The edge switch corresponds to the storage server of the data center, and is used to complete the check of whether the data block of the uploaded file has been stored by the storage server, the AES encryption operation of the unstored plaintext data block, and the AES decryption operation of the encrypted data block of the downloaded file;

[0011] The centralized controller is located inside the data center and is responsible for key negotiation with the controller inside the local area network. When a file is uploaded, if a new file data block is stored, a new table entry is added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file data block is no longer used by other files, an old table entry is deleted from the hash-index table of the corresponding storage server.

[0012] In a second aspect, the present invention provides a method for uploading files from a local area network client using the data deduplication system as described in the first aspect, comprising the following steps:

[0013] Step 1: When uploading a file, the LAN client calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash value of m data blocks and is sent to the corresponding storage server through the network security IP tunnel.

[0014] Step 2: Each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has been stored, its index information on the storage server is recorded;

[0015] Step 3: The storage server integrates the received multiple query result data packets and returns a complete query result data packet;

[0016] Step 4: After receiving the complete query result data packet, the LAN client sends the missing data blocks to the storage server;

[0017] Step 5: After the missing data block passes through the edge switch, the hash value of the plain text is calculated and then encrypted through AES. The encrypted data block of the ciphertext is sent to the storage server;

[0018] Step 6: The storage server receives the data block that has been encrypted by the edge switch, updates the data block counter maintained by the storage server, and sends the saved hash value and index to the centralized controller, thereby updating the hash-index table of the corresponding edge switch.

[0019] Optionally, in the third step, the complete query result data packet includes a 01 bit string, and the corresponding position is set to 0 to indicate that the corresponding data block is not stored, and the corresponding position is set to 1 to indicate that the corresponding data block has been stored on the server.

[0020] Optionally, in the fourth step, after receiving the complete query result data packet, the local area network client sends the missing data blocks with corresponding positions set to 0 to the storage server.

[0021] In a third aspect, the present invention provides a method for downloading files from a local area network client using the data deduplication system as described in the first aspect, comprising the following steps:

[0022] Step 1: The LAN client sends the file ID to be downloaded;

[0023] Step 2: The storage server sends the encrypted ciphertext to the LAN client through the Internet security IP tunnel according to the index list of the corresponding files;

[0024] Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted by AES to obtain the original plaintext of the data.

[0025] In a fourth aspect, the present invention proposes a method for deleting client files in a local area network using the data deduplication system as described in the first aspect, comprising the following steps:

[0026] Step 1: The LAN client sends the file ID to be deleted;

[0027] Step 2: The storage server decrements the corresponding data block counter by 1 in turn according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the centralized controller is notified to update the hash-index table in the edge switch.

[0028] The beneficial effects of the present invention are as follows: the present invention proposes a system and method for establishing an inter-network secure IP tunnel and performing data deduplication on an online programmable switch, so that deduplication can be performed using a finer data block size granularity, greatly improving the efficiency of data deduplication, thereby being able to better identify the information redundancy between files uploaded by multiple users, reduce the storage overhead of the storage server, and reduce costs. And by establishing a secure IP tunnel, it is ensured that the programmable switch can parse and process plain text data, so there is no need to introduce other additional storage data like the existing convergence encryption method. The present invention also saves the CPU overhead of the corresponding storage server by offloading the hash-index table lookup and AES encryption and decryption operations to the edge programmable switch, thereby improving CPU resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a system framework diagram of a data deduplication system based on an internetwork secure IP tunnel. DETAILED DESCRIPTION

[0030] The present invention will now be described in further detail with reference to the accompanying drawings.

[0031] Embodiment 1

[0032] This embodiment proposes a data deduplication system based on an inter-network secure IP tunnel. Figure 1 This is a schematic diagram of the overall system framework. As shown in the figure, it is mainly divided into two parts: LAN client and data center. The sender includes multiple clients and port programmable switches; the receiver is divided into core programmable switches, aggregation programmable switches, edge programmable switches and multiple storage servers. The LAN client needs to upload, download and delete files. An IP security tunnel is established between the LAN port switch and the core switch of the data center. The edge switch performs hash index table lookup and AES algorithm operations.

[0033] The data deduplication system based on inter-network secure IP tunnel specifically includes: an inter-network secure IP tunnel established between the local area network and the data center based on a programmable switch; an edge switch corresponding to the storage server, which is used to complete the on-line calculation of data deduplication and encryption; and a centralized controller within the data center network.

[0034] The network security IP tunnel module encrypts the entire data packet related to data deduplication leaving the local area network or data center and decrypts the entire data packet related to data deduplication entering the local area network or data center.

[0035] The edge switch completes the check on whether the corresponding data block of the uploaded file has been stored by the storage server, performs AES encryption operation on the plaintext data block that has not been stored, and decrypts the encrypted data block of the downloaded file.

[0036] The centralized controller is responsible for key negotiation with the controller within the LAN. When a file is uploaded, if a new file block is stored, a new entry needs to be added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file block is no longer used by other files, an old entry needs to be deleted from the hash-index table of the corresponding storage server.

[0037] The system establishes a secure IP tunnel between the local area network and the endpoint programmable switch of the data center. The endpoint switch decrypts the ciphertext data packets received from the external network and encrypts the plaintext data packets sent to the external network, thereby ensuring that the data packets are in plaintext form inside the local area network and the data center. The main deduplication function is unloaded to the edge programmable switch in the data center network. Before the local area network client sends the actual plaintext data, it calculates the corresponding hash code and sends the query data packet. After the data packet arrives at the edge switch of the data center, it queries whether the corresponding entry in the hash-index table of the programmable switch exists, thereby determining whether the corresponding storage server has stored the data block. Only when the data block is found to be missing, the client in the local area network will subsequently upload the corresponding data and encrypt the data block through the programmable switch in the data center network. When the client in the local area network needs to download the file, the storage ciphertext sent by the storage server is decrypted by the edge programmable switch in the data center network. Therefore, the system can support finer deduplication granularity, thereby significantly improving the efficiency of deduplication while introducing acceptable additional storage costs and reducing server CPU consumption.

[0038] Embodiment 2

[0039] This embodiment proposes a method for uploading, downloading and deleting files using the data deduplication system based on the network security IP tunnel of the first embodiment, which specifically includes the following steps.

[0040] For file upload operations:

[0041] In the first step, when uploading a file, the LAN client calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash value of m data blocks and is sent to the corresponding storage server through the IP security tunnel.

[0042] In the second step, each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has been stored, its index information on the storage server is recorded;

[0043] In the third step, the storage server integrates the multiple query result data packets received and returns a complete data packet, which contains a 01 bit string, and sets the corresponding position to 0 to indicate that the corresponding data block is not stored, and sets the corresponding position to 1 to indicate that the corresponding data block has been stored on the server;

[0044] Step 4: After receiving the complete query result data packet, the LAN client sends the missing data blocks with the corresponding positions set to 0 to the storage server;

[0045] Step 5: After the missing data block passes through the edge switch, the hash value of the plain text is calculated and then encrypted by AES to obtain the encrypted data block of the ciphertext and send it to the storage server;

[0046] In the sixth step, the storage server receives the data block that has been encrypted by the edge switch, updates the data block counter maintained by the storage server, and sends the saved hash value and index to the controller of the data center, thereby updating the hash-index table of the corresponding programmable switch.

[0047] For file download operations:

[0048] Step 1: The LAN client sends the file ID to be downloaded;

[0049] Step 2: The storage server sends the encrypted ciphertext to the LAN client through the IP security tunnel according to the index list of the corresponding files;

[0050] Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted by AES to obtain the original plaintext of the data.

[0051] For file deletion operations:

[0052] Step 1: The LAN client sends the file ID to be deleted;

[0053] Step 2: The storage server decrements the corresponding data block counter by 1 in turn according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the controller is notified to update the hash-index table in the edge switch.

[0054] The innovation of the present invention lies in that the system and method complete the inter-network IP security tunnel for the first time through the programmable switch on the network and offload the hash index table lookup for data deduplication and the plaintext AES encryption operation and ciphertext AES decryption operation to the programmable switch inside the data center, thereby ensuring that the data is encrypted and transmitted in an external insecure network environment, and the plaintext is parsed and processed inside the local area network and the data center, so that more efficient deduplication can be performed with a smaller data block size without introducing additional storage overhead. At the same time, the present invention can offload the computing tasks of the storage server side to the corresponding edge switch, and use the linear processing speed of the programmable switch to reduce the CPU overhead of the storage server side.

[0055] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A data deduplication system based on an internetwork secure IP tunnel, characterized in that: include: Inter-network secure IP tunnels, edge switches and centralized controllers; The inter-network secure IP tunnel is established based on a programmable switch, which is located between the local area network and the data center, and is used to encrypt the entire packet of data packets related to data deduplication leaving the local area network and the data center, and to decrypt the entire packet of data packets related to data deduplication entering the local area network and the data center; The edge switch corresponds to the storage server of the data center, and is used to complete the check of whether the data block of the uploaded file has been stored by the storage server, the AES encryption operation of the unstored plaintext data block, and the AES decryption operation of the encrypted data block of the downloaded file; The centralized controller is located inside the data center and is responsible for key negotiation with the controller inside the local area network. When a file is uploaded, if a new file data block is stored, a new table entry is added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file data block is no longer used by other files, an old table entry is deleted from the hash-index table of the corresponding storage server.

2. A method for uploading files from a local area network client using the data deduplication system as claimed in claim 1, characterized in that: The steps include: Step 1: When uploading a file, the LAN client calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash value of m data blocks and is sent to the corresponding storage server through the network security IP tunnel. Step 2: Each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has been stored, its index information on the storage server is recorded; Step 3: The storage server integrates the received multiple query result data packets and returns a complete query result data packet; Step 4: After receiving the complete query result data packet, the LAN client sends the missing data blocks to the storage server; Step 5: After the missing data block passes through the edge switch, the hash value of the plain text is calculated and then encrypted through AES. The encrypted data block of the ciphertext is sent to the storage server; Step 6: The storage server receives the data block that has been encrypted by the edge switch, updates the data block counter maintained by the storage server, and sends the saved hash value and index to the centralized controller, thereby updating the hash-index table of the corresponding edge switch.

3. The method for uploading files from a local area network client as claimed in claim 2, characterized in that: In the third step, the complete query result data packet includes a 01 bit string, and the corresponding position is set to 0 to indicate that the corresponding data block has not been stored, and the corresponding position is set to 1 to indicate that the corresponding data block has been stored on the server.

4. The method for uploading files from a local area network client as claimed in claim 3, characterized in that: In the fourth step, after receiving the complete query result data packet, the local area network client sends the missing data blocks with corresponding positions set to 0 to the storage server.

5. A method for downloading files from a local area network client using the data deduplication system as claimed in claim 1, characterized in that: The steps include: Step 1: The LAN client sends the file ID to be downloaded; Step 2: The storage server sends the encrypted ciphertext to the LAN client through the Internet security IP tunnel according to the index list of the corresponding files; Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted by AES to obtain the original plaintext of the data.

6. A method for deleting client files in a local area network using the data deduplication system as claimed in claim 1, characterized in that: The steps include: Step 1: The LAN client sends the file ID to be deleted; Step 2: The storage server decrements the corresponding data block counter by 1 in turn according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the centralized controller is notified to update the hash-index table in the edge switch.

Images