Small-file data storage method, small-file data reading method, small-file data storage device and small-file data reading device in distributed file system
A distributed file and data reading technology, which is applied in the field of reading and small file data storage, can solve the problems of small file storage performance response speed impact, high access frequency and large access volume of small files, etc., to reduce pressure, The effect of speeding up efficiency and reducing response time
Inactive Publication Date: 2017-06-13
北京思特奇信息技术股份有限公司
View PDF9 Cites 16 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] Small file access has the characteristics of high access frequency and large amount of access. The access of small files in the traditional file system puts a lot of pressure on the host disk. The distributed file system provides external services in the form of clusters, but the storage performance of small files is still affected by network time. Latency and disk response speed
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0029] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.
[0030] see figure 1 , is a method for storing small file data in a distributed file system of embodiment 1, comprising:
[0031] S1a, in the distributed file system, the distributed memory library is used as a hierarchical cache layer, and an asynchronous commit cache module is added;
[0032] S2a, when writing the small file data into the distributed file system, first write the small file data into the hierarchical cache layer;
[0033] S3a, when the amount of small file data in the hierarchical cache layer reaches a preset amount, the asynchronous commit cache module reads the small file data from the hierarchical cache layer and writes it into the data persistence layer;
[0034] S4a, after the asynchronous su...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a small-file data storage method, a small-file data reading method, a small-file data storage device and a small-file data reading device in a distributed file system. In the distributed file system, distributed memories serve as graded buffering layers, and asynchronously submitted buffering modules are added. When written into the distributed file system, data are written into the graded buffering layers firstly to reduce pressure on a persistent-layer magnetic disk through a lightweight buffering function of the graded buffering layers, and then, the asynchronously submitted buffering modules read small-file data from the graded buffering layers and store the data to a data persistent layer to shorten small-file data persistence response time, so that storage efficiency of the small-file data in the distributed file system is improved.
Description
technical field [0001] The invention relates to the technical field of data storage and reading, in particular to a method and device for storing and reading small file data in a distributed file system. Background technique [0002] The storage of massive small files (LOSF for short, lots of small files) has been a difficult problem in the industry since its appearance. Through the research, testing and use of several distributed storage systems, compared with other distributed systems, the storage of massive small files is more focused on solving two problems: (1) metadata information organization and management of massive small files; (2) ) The read and write speed of massive small files. [0003] Small file access has the characteristics of high access frequency and large amount of access. The access of small files in the traditional file system puts a lot of pressure on the host disk. The distributed file system provides external services in the form of clusters, but t...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06G06F17/30
CPCG06F3/0611G06F3/0656G06F3/067G06F16/13G06F16/182
Inventor 王敏
Owner 北京思特奇信息技术股份有限公司