NoSQL-based massive small file storage structure for aviation logistics and storage method of NoSQL-based massive small file storage structure

Abstract

The invention discloses an NoSQL-based massive small file storage structure for aviation logistics and a storage method of the NoSQL-based massive small file storage structure. The NoSQL-based massive small file storage structure for aviation logistics comprises a data cache region, a data active region and a data permanent region. The file storage method of the NoSQL-based massive small file storage structure for aviation logistics comprises three processes including file storage, file access and permanent file processing. According to the NoSQL-based massive small file storage structure for aviation logistics and the storage method of the NoSQL-based massive small file storage structure, a multi-stage storage mechanism is designed, a file pre-storing mechanism is adopted, accordingly, the storage efficiency of small files is effectively improved, the memory consumption of a cluster NameNode is remarkably reduced, and the utilization rate of a disk is effectively improved.

Description

technical field [0001] The invention belongs to the technical field of data storage, and in particular relates to a NoSQL-based massive aviation logistics small file storage structure and an access method thereof. Background technique [0002] For the aviation logistics industry, since most information exchange exists in the form of messages, and there are many types of messages, there are three main types: xml, cimp, typeb, of which xml type data accounts for 80% More than %, the size of each message is basically about 1-3kB. The daily data exchange volume on the iLink platform of the aviation logistics information system is about 10G. For the data of 10G size, there are about tens of millions of messages. With the continuous development of the civil aviation information industry, massive data information is growing explosively. The traditional data storage methods can no longer meet the needs of the current industry. There are two main reasons: one is that the storage eff...

AI Technical Summary

Problems solved by technology

With the continuous development of the civil aviation information industry, massive data information is growing explosively. The traditional data storage methods can no longer meet the needs of the current industry. There are two main reasons: one is that the storage efficiency is not high, and the other is the response The time is too long; therefore, in order to meet the needs of production, each enterprise unit gradually connects the data to the cloud storage platform for storage

[0003] At present, the most typical cloud storage platform is Hadoop Distributed File System (HDFS). Compared with traditional storage media, Hadoop Distributed File System (HDFS) supports large-scale file storage through multiple cheap machines, and has strong scalability. It solves the limitation problem of storage space; at the same time, HDFS can provide high-throughput data access, which is very suitable for applications on large-scale data sets, and can ensure the reliability of data storage even in the case of errors; however, HDFS It is a distributed system specially designed for large file storage, and the efficiency of storing small files is extremely low. When HDFS stores small files, the distributed cluster NameNode will consume a large amount of memory space to manage the metadata of each small file, making file access The efficiency is greatly reduced. In addition, HDFS storage is based on data blocks. A large number of small file storage will greatly waste disk space, resulting in low storage efficiency.

Images