Data deduplication system and method based on internet security IP tunnel

Abstract

The application provides a data deduplication system and method based on an Internet security IP tunnel, which comprises: an Internet security IP tunnel based on a programmable switch between a local area network and a data center, which is used for encrypting the whole data packet of data deduplication related data packets leaving the local area network and the data center and decrypting the whole data packet of data deduplication related data packets entering the local area network and the data center; an edge switch corresponding to a storage server, which is used for completing in-network computing of data deduplication and encryption; and a centralized controller inside the data center network, which is responsible for key negotiation with a controller inside the local area network. The application can support a finer deduplication granularity, so that the efficiency of the data deduplication can be significantly improved while introducing acceptable additional storage costs and reducing the consumption of the server CPU.

Description

Technical Field

[0001] This invention belongs to the field of network systems, specifically relating to a data deduplication system and method based on secure IP tunnels between networks. Background Technology

[0002] With the continuous development of internet technology and the ever-growing number of internet users, the total amount of data globally has surged dramatically. In this context, managing data storage economically and efficiently has become one of the most challenging and crucial tasks in massive storage systems during the big data era. Meanwhile, limited by the storage capacity of individual devices, individuals and organizations often rely on cloud storage service providers to achieve low-cost storage, transmission, and backup of their ever-increasing data. To further improve data storage efficiency and reduce costs, common cloud service providers typically employ data deduplication technologies to minimize the repeated storage of duplicate data, thereby reducing data storage overhead and also decreasing upload bandwidth for users.

[0003] Furthermore, considering the presence of numerous malicious attackers in the real-world network environment, it is often necessary to encrypt the plaintext data before transmission to prevent the leakage of users' private data. However, this can lead to different users using different private keys to encrypt the same data, resulting in the same plaintext ultimately becoming different ciphertexts, thus making it impossible for the data storage server to detect data duplication.

[0004] Current network research employs two main approaches: convergent encryption and SSL / TLS-based methods. For plaintext M, convergent encryption first calculates a hash value H(M) using a hash function, then uses this hash value as the encryption key to encrypt the plaintext, resulting in E(H(M), M). To ensure users can still decrypt the data during download, the data center also needs to store this hash value H(M) as the encryption key. For security reasons, the hash value H(M) is encrypted using each user's private key, resulting in E(Ka, H(M)). SSL / TLS-based methods, on the other hand, encrypt the data directly during transmission using the SSL or TLS protocol, and then decrypt it upon receipt by the data storage server using the same protocol before further processing. For convergent encryption methods, because the encryption result of the hash key needs to be stored separately, the block size for data deduplication can only be at the file level or a relatively large block size (KB level). Otherwise, when the block size is very small, the additional storage overhead cannot be ignored, resulting in the inability to fully utilize redundant information in the file for deduplication, leading to poor deduplication performance. For SSL or TLS methods, since the data server uses plaintext for data deduplication checks, storing user privacy data directly in plaintext would pose significant security risks. Therefore, data must still be encrypted before each storage and decrypted each time it is downloaded, increasing server-side CPU overhead.

[0005] Therefore, providing a system / method that can perform deduplication with a smaller block size and offload complex encryption operations from the server to on-net computing is a problem that urgently needs to be solved by those skilled in the art. Such a system / method can significantly increase the deduplication efficiency of data storage systems and significantly reduce the CPU overhead of the server. Summary of the Invention

[0006] This invention addresses the shortcomings of existing storage server file transfer data deduplication methods, which cannot simultaneously balance data deduplication efficiency and storage server CPU overhead. It provides a data deduplication system and method based on secure inter-network IP tunnels. By establishing encrypted inter-network IP tunnels, it ensures the security and reliability of data packets transmitted between the local area network and the data center network. This allows for the preservation of user privacy data in untrusted transmission environments with malicious nodes, while enabling the data center network to obtain the plaintext of user data. This facilitates subsequent data analysis and processing without introducing additional storage overhead. Furthermore, the encryption and decryption operations of stored data are offloaded to edge programmable switches, thereby significantly improving deduplication efficiency and server CPU utilization, saving data center storage server costs.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a data deduplication system based on an inter-network secure IP tunnel, comprising: an inter-network secure IP tunnel, an edge switch, and a centralized controller;

[0009] The secure IP tunnel between networks is established based on a programmable switch and is located between the local area network (LAN) and the data center. It is used to encrypt the entire data packets related to data deduplication leaving the LAN and the data center and to decrypt the entire data packets related to data deduplication entering the LAN and the data center.

[0010] The edge switch corresponds to the storage server in the data center and is used to check whether the data blocks of the uploaded file have been stored by the storage server, perform AES encryption on the plaintext data blocks that have not been stored, and perform AES decryption on the encrypted data blocks of the downloaded file.

[0011] The centralized controller is located inside the data center and is responsible for key negotiation with the controllers inside the local area network. When a file is uploaded, if a new file data block is stored, a new entry is added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file data block is no longer used by other files, an old entry is deleted from the hash-index table of the corresponding storage server.

[0012] Secondly, the present invention provides a method for uploading files to a local area network client using the data deduplication system as described in the first aspect, comprising the following steps:

[0013] Step 1: When uploading files, the LAN client calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash values ​​of m data blocks and is sent to the corresponding storage server through a secure IP tunnel.

[0014] Step 2: Each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has already been stored, its index information on the storage server is recorded.

[0015] Step 3: The storage server integrates the multiple query result data packets received and returns a complete query result data packet;

[0016] Step 4: After receiving the complete query result data packet, the LAN client sends the missing data block to the storage server;

[0017] Step 5: After the corresponding missing data block passes through the edge switch, the hash value of the plaintext is calculated and then encrypted with AES to obtain the encrypted data block of ciphertext, which is then sent to the storage server.

[0018] Step 6: The storage server receives the data block that has been encrypted by the edge switch, updates the data block counter maintained by the storage server, and sends the saved hash value and index to the centralized controller, thereby updating the hash-index table of the corresponding edge switch.

[0019] Optionally, in the third step, the complete query result data packet contains a 01 bit string, and the corresponding position is set to 0 to indicate that the corresponding data block is not stored, and the corresponding position is set to 1 to indicate that the corresponding data block has been stored on the server.

[0020] Optionally, in the fourth step, after receiving the complete query result data packet, the local area network client sends the missing data block with the corresponding position set to 0 to the storage server.

[0021] Thirdly, the present invention provides a method for downloading client files on a local area network using a data deduplication system as described in the first aspect, comprising the following steps:

[0022] Step 1: The LAN client sends the file ID to be downloaded;

[0023] Step 2: The storage server sends the encrypted ciphertext to the local area network client through a secure IP tunnel based on the index list of the corresponding files.

[0024] Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted using AES to obtain the original plaintext data.

[0025] Fourthly, the present invention proposes a method for deleting client files on a local area network using a data deduplication system as described in the first aspect, comprising the following steps:

[0026] Step 1: The LAN client sends the file ID to be deleted;

[0027] Step 2: The storage server decrements the corresponding data block counter by 1 according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the centralized controller is notified to update the hash-index table in the edge switch.

[0028] The beneficial effects of this invention are as follows: This invention proposes a system and method for establishing a secure IP tunnel between network programmable switches and performing data deduplication. This allows for deduplication with finer data block size granularity, significantly improving deduplication efficiency. This, in turn, enables better identification of information redundancy between files uploaded by multiple users, reduces storage overhead on storage servers, and lowers costs. Furthermore, by establishing a secure IP tunnel, the programmable switch can process data in plaintext, thus eliminating the need for additional data storage as with existing convergent encryption methods. This invention also offloads hash-index lookup and AES encryption / decryption operations to the edge programmable switch, thereby saving CPU overhead on the corresponding storage servers and improving CPU resource utilization. Attached Figure Description

[0029] Figure 1 This is a system framework diagram of a data deduplication system based on secure IP tunnels between networks. Detailed Implementation

[0030] The invention will now be described in further detail with reference to the accompanying drawings.

[0031] Example 1

[0032] This embodiment proposes a data deduplication system based on secure IP tunnels between networks. Figure 1 This is a schematic diagram of the overall system framework. As shown, it is mainly divided into two parts: the LAN client and the data center. The sending side includes multiple clients and port programmable switches; the receiving side is further divided into core programmable switches, aggregation programmable switches, edge programmable switches, and multiple storage servers. The LAN clients need to perform file upload, download, and deletion operations. An IP secure tunnel is established between the LAN port switches and the core switch in the data center, and the edge switches perform hash index lookup and AES algorithm operations.

[0033] The data deduplication system based on secure IP tunnels between networks specifically includes: a secure IP tunnel between a local area network and a data center established using a programmable switch; an edge switch corresponding to the storage server, used for on-network computation to perform data deduplication and encryption; and a centralized controller within the data center network.

[0034] The Internet Security IP Tunnel module encrypts all data packets related to data deduplication when leaving the local area network or data center, and decrypts all data packets related to data deduplication when entering the local area network or data center.

[0035] The edge switch performs a check to see if the data blocks corresponding to the uploaded file have been stored on the storage server, performs AES encryption on plaintext data blocks that have not been stored, and decrypts the encrypted data blocks of the downloaded file.

[0036] The centralized controller is responsible for key negotiation with the controllers within the local area network. When a file is uploaded, if a new file block is stored, a new entry needs to be added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file block is no longer used by other files, an old entry needs to be deleted from the hash-index table of the corresponding storage server.

[0037] This system establishes a secure IP tunnel between the local area network (LAN) and the endpoint programmable switches in the data center. The endpoint switches decrypt ciphertext data packets received from the external network and encrypt plaintext data packets sent to the external network, ensuring that data packets are in plaintext form both within the LAN and the data center. The primary deduplication function is offloaded to an edge programmable switch within the data center network. Before sending plaintext data, LAN clients calculate the corresponding hash code and send a query data packet. Upon reaching the edge switch in the data center, the switch checks the hash-index table to determine if the corresponding data block is already stored on the storage server. Only when a missing data block is detected will the LAN client subsequently upload the data, and the data block will be AES encrypted by the programmable switch within the data center network. When a client in the LAN needs to download a file, the ciphertext sent by the storage server is decrypted by the edge programmable switch within the data center network. Therefore, this system supports finer-grained deduplication, significantly improving deduplication efficiency while introducing acceptable additional storage costs and reducing server CPU consumption.

[0038] Example 2

[0039] This embodiment proposes a method for uploading, downloading, and deleting files using the data deduplication system based on the secure IP tunnel in Embodiment 1, specifically including the following steps.

[0040] For file upload operations:

[0041] The first step is that when the LAN client uploads a file, it calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash values ​​of m data blocks and is sent to the corresponding storage server through an IP security tunnel.

[0042] The second step is that each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has already been stored, its index information on the storage server is recorded.

[0043] The third step is for the storage server to integrate the multiple query result data packets received and return a complete data packet. The data packet contains a 01 bit string, and the corresponding position is set to 0 to indicate that the corresponding data block has not been stored, and the corresponding position is set to 1 to indicate that the corresponding data block has been stored on the server.

[0044] Fourth, after receiving the complete query result data packet, the LAN client sends the missing data blocks with the corresponding position set to 0 to the storage server;

[0045] The fifth step involves the corresponding missing data block passing through the edge switch, where the plaintext is first hashed and then encrypted using AES to obtain the encrypted data block, which is then sent to the storage server.

[0046] The sixth step involves the storage server receiving data blocks that have been encrypted by the edge switch, updating the data block counter maintained by the storage server, and sending the saved hash value and index to the data center controller, thereby updating the hash-index table of the corresponding programmable switch.

[0047] For file download operations:

[0048] Step 1: The LAN client sends the file ID to be downloaded;

[0049] Step 2: The storage server sends the encrypted ciphertext to the local area network client through an IP secure tunnel according to the index list of the corresponding files.

[0050] Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted using AES to obtain the original plaintext data.

[0051] For file deletion operations:

[0052] Step 1: The LAN client sends the file ID to be deleted;

[0053] Step 2: The storage server decrements the corresponding data block counter by 1 according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the controller is notified to update the hash-index table in the edge switch.

[0054] The innovation of this invention lies in the fact that this system and method, for the first time, completes the inter-network IP secure tunneling through a network-connected programmable switch and offloads the hash index lookup for data deduplication, as well as the plaintext AES encryption and ciphertext AES decryption operations, to a programmable switch inside the data center. This ensures that data is encrypted during transmission in insecure external network environments and processed in plaintext within the local area network and data center. This allows for more efficient deduplication with smaller data block sizes and avoids introducing additional storage overhead. Furthermore, this invention can offload the computing tasks of the storage server to corresponding edge switches and utilize the linear processing speed of the programmable switches to reduce the CPU overhead of the storage server.

[0055] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A data deduplication system based on secure IP tunnels across networks, characterized in that, include: Secure IP tunnels between networks, edge switches, and centralized controllers; The secure IP tunnel between networks is established based on a programmable switch and is located between the local area network (LAN) and the data center. It is used to encrypt the entire data packets related to data deduplication leaving the LAN and the data center and to decrypt the entire data packets related to data deduplication entering the LAN and the data center. The edge switch corresponds to the storage server in the data center and is used to check whether the data blocks of the uploaded file have been stored by the storage server, perform AES encryption on the plaintext data blocks that have not been stored, and perform AES decryption on the encrypted data blocks of the downloaded file. The centralized controller is located inside the data center and is responsible for key negotiation with the controllers inside the local area network. When a file is uploaded, if a new file data block is stored, a new entry is added to the hash-index table of the corresponding storage server. When a file is deleted, if the stored file data block is no longer used by other files, an old entry is deleted from the hash-index table of the corresponding storage server.

2. The local area network client file upload method using the data deduplication system as described in claim 1, characterized in that, Includes the following steps: Step 1: When uploading files, the LAN client calculates the hash value of the corresponding data block and assembles it into multiple query data packets in sequence. Each data packet contains the hash values ​​of m data blocks and is sent to the corresponding storage server through a secure IP tunnel. Step 2: Each query data packet sent by the LAN client completes the query function when it passes through the edge switch corresponding to the storage server. If the data block has already been stored, its index information on the storage server is recorded. Step 3: The storage server integrates the multiple query result data packets received and returns a complete query result data packet; Step 4: After receiving the complete query result data packet, the LAN client sends the missing data blocks to the storage server; Step 5: After the corresponding missing data block passes through the edge switch, the hash value of the plaintext is calculated and then encrypted with AES to obtain the encrypted data block of ciphertext, which is then sent to the storage server. Step 6: The storage server receives the data block that has been encrypted by the edge switch, updates the data block counter maintained by the storage server, and sends the saved hash value and index to the centralized controller, thereby updating the hash-index table of the corresponding edge switch.

3. The local area network client file upload method as described in claim 2, characterized in that: In the third step, the complete query result data packet contains a 01 bit string, and the corresponding position is set to 0 to indicate that the corresponding data block has not been stored, and the corresponding position is set to 1 to indicate that the corresponding data block has been stored on the server.

4. The local area network client file upload method as described in claim 3, characterized in that: In the fourth step, after receiving the complete query result data packet, the local area network client sends the missing data block with the corresponding position set to 0 to the storage server.

5. A method for downloading client files on a local area network using the data deduplication system as described in claim 1, characterized in that, Includes the following steps: Step 1: The LAN client sends the file ID to be downloaded; Step 2: The storage server sends the encrypted ciphertext to the local area network client through a secure IP tunnel based on the index list of the corresponding files. Step 3: When the encrypted ciphertext passes through the corresponding edge switch, it is decrypted using AES to obtain the original plaintext data.

6. The method for deleting client files on a local area network using the data deduplication system as described in claim 1, characterized in that, Includes the following steps: Step 1: The LAN client sends the file ID to be deleted; Step 2: The storage server decrements the corresponding data block counter by 1 according to the index list of the corresponding file. If the corresponding counter is reduced to 0, the centralized controller is notified to update the hash-index table in the edge switch.

Images