Real-time repetition removal and transmission method for data in network file system

Abstract

The invention relates to a real-time repetition removal and transmission method for data in a network file system, and belongs to the technical field of computer data storage. The method comprises the following steps of: setting a file metadata table, a data block index table and a file composition table in a client and a server respectively, and setting a to-be-transmitted message queue for storing data and updating message in the client; receiving and responding an operating command initiated to the network file system by a client application program through a file system drive by the client, wherein the operating command comprises the operation of creating a new file, writing data into an existing file, reading the data from the existing file and deleting the existing file; and settinga network service interface for uploading and downloading data block contents and receiving and answering client message in the server. The method can delete the repeated data so as to save the storage space, avoid transmitting the existing data of the opposite side between the client and the server and reduce the overhead of network bandwidth; meanwhile, the method supports a file blocking method of fixed length and unfixed length so as to improve the utilization rate of the storage space.

Description

technical field [0001] The invention relates to a real-time data deduplication and transmission method in a network file system, belonging to the technical field of computer data storage. Background technique [0002] For a long time, how to store as much data as possible with as little space as possible and how to transmit as much information as possible with as little network bandwidth overhead as possible has been the core issue in the field of network storage. Reducing the storage and transmission of duplicate data in files is the key technology to solve the above problems. [0003] Data deduplication technology, also known as data deduplication technology, emerged at the beginning of this century and has been popularized in recent years. The idea of ​​data deduplication is not complicated: first discover the same data units that appear in different files, different versions of the same file, or even in different locations of the same version, and then index the data un...

AI Technical Summary

Problems solved by technology

However, in its method, only after the file is closed, the whole file is divided into blocks and synchronized to the server side. At least the following significant disadvantages exist in its method: 1. Data block and deduplication are not carried out in real time. The new data at the end cannot be transmitted to the server in time; second, the cutting efficiency is low, even if only a small part of the content in the file is modified, the entire file needs to be cut again; third, when opening a new file, you need to first All the data blocks of the file are spliced ​​into a temporary file before the file can be operated on, so it takes a long time to wait

[0005] Other existing network systems, such as Network File System (NFS) and Server Message Block (SMB), do not use data deduplication technology, nor can they provide functions to reduce storage space occupation and network data transmission

Images