On-line recovering method of junk metadata in distributed file system
A technology of distributed files and metadata, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of service processing impact, waste of CPU resources, etc., and achieve the effect of saving storage resources and reducing load
Active Publication Date: 2011-04-20
DAWNING INFORMATION IND BEIJING +1
View PDF4 Cites 17 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
This method needs to scan the entire namespace. When there are many subitems in the namespace, it will waste CPU resources. During this period, other service processing will be affected.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0016] (1) Index nodes are numbered with 64-bit unsigned integers, starting from 0, and gradually accumulating when assigning index node numbers. Since the range of numbers that can be expressed by 64-bit unsigned integers is very large, assuming that 10,000 index nodes are created in one second, it will take about 58494243 years for all 64 unsigned numbers to be used up, so it can be approximated that the number of index nodes is infinite . In the invention, the metadata of every 32K index nodes is stored in the same file, and all index node files are stored on the disk in a two-level tree structure directory according to the node numbers.
[0017] (2) Store the directory structure of the index node file according to figure 1 Organized into a multi-way search tree, each non-leaf node of the tree represents a subdirectory, while the leaf node represents the index node file, and each tree node is represented by a specific data structure. When deleting a file or directory, fir...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention discloses an-online recovering method of junk metadata in a distributed file system. In the method, a storage structure of the metadata on a local file system is snapshoted by a multipath search tree; when file or catalogue actions are processed and deleted, a junk index node is not cleared synchronously, and tree nodes describing the index node files and all ancestor nodes thereof are marked. When the number of the junk index nodes reaches to a threshold value, an asynchronous recycling thread is waked up; the thread traverses the multipath search tree; whether the nodeor sub-node thereof has the junk metadata or not is determined by the junk marker bit, and the junk is further cleared. The method disclosed by the invention can recover the invalid data generated during the running of the file system under the condition that the distributed file system supplies service outwardly, so that the storage resource on a metadata server is saved, and the load when the metadata sever accesses to the local file system is reduced.
Description
technical field [0001] The invention relates to file management in a distributed file system, in particular to an online recycling method for garbage metadata in a distributed file system. Background technique [0002] In a distributed file system, an efficient metadata management method is an important factor that determines its overall performance. The metadata management of modern distributed file systems is generally based on the local file system, which can reduce the cost of development, debugging and maintenance. For distributed file systems that store massive files and directories, the amount of metadata is huge. In order to improve memory access efficiency, metadata is usually grouped according to node numbers and stored in hierarchical directories, such as figure 1 shown. The bottom node of the tree structure shown in the figure represents the index node file storing metadata, and data of multiple index nodes is stored in one file, and the node numbers of these i...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F17/30
Inventor 杨浩邵宗有苗艳超王勇马照云
Owner DAWNING INFORMATION IND BEIJING