SThe invention discloses a storage and reading method of a small file storage architecture
A storage architecture and small file technology, applied in the field of big data, can solve the problems of low reading efficiency, low storage and reading efficiency, achieve high storage and reading efficiency, reduce the process of metadata information management, and read Take the effect of high efficiency
Inactive Publication Date: 2019-04-12
安徽云融信息技术有限公司
View PDF2 Cites 2 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] The object of the present invention is to provide a storage and reading method of a small file storage architecture, by directly storing the small file in the Hbase table structure when the file is written, and using the Hbase self-indexing mechanism when reading, It solves the problem of low reading efficiency, and compared with HDFS, it solves the process of metadata information management for a large number of small files, and the problem of low storage and reading efficiency
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
[0034] see figure 1 As shown, a kind of small file storage framework of the present invention, storage framework comprises user layer, access layer, storage layer from top to bottom, and storage layer comprises Hbase storage and HDFS storage;
[0035] The user layer uses the HTTP protocol or the HTTPS protocol to interact with the access layer;
[0036] The access layer includes two types of servers, UP / Down Sever and CGI Proxy Sever, and the Hbase file interface;...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a storage and reading method for a small file storage architecture, and relates to the technical field of big data. The storage architecture comprises a user layer, an access layer and a storage layer, The storage method comprises the steps that S01, a user uploads a to-be-stored file; S; s02, judging the size of the file, and when the size of the file is larger than a setthreshold value, turning to the step S04; O; otherwise, going to the step S03; S; s03, further analyzing the file request; S; s04, calling an HDFS file write-in interface for operation; S; s05, writing the file information into a corresponding table in an Hbase storage; according to the method, when the file is written in, the small file is directly stored in the Hbase table structure, during reading, the reading efficiency is high through the Hbase index mechanism, compared with an HDFS, the process of metadata information management of a large number of small files is reduced, and the storage and reading efficiency is high.
Description
technical field [0001] The invention belongs to the technical field of big data, and in particular relates to a storage and reading method of a small file storage architecture. Background technique [0002] With the continuous development of the Internet and the explosive growth of digital information, how to efficiently process and store massive data has become an urgent problem to be solved. With its open source, . With its easy-to-use features, the Hadoop distributed platform has become the preferred solution for managing and processing massive amounts of data. With the Hadoop distributed file system as the basic storage, and providing various services through the MapReduce computing framework, users can build Hadoop clusters in a low-cost hardware environment, and use Hbase and other components to complete massive data processing and analysis tasks, thereby providing large data sets It provides a low-cost distributed storage solution for processing and applications. ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/13G06F16/182
Inventor 胡翔
Owner 安徽云融信息技术有限公司