Massive data storage and retrieval method

Abstract

The invention discloses a massive data storage and retrieval method comprising the following steps of: (1), in a file system of a server, building directories according to periods of time, wherein each directory contains multiple equipment data files of (id).data command; (2), after the server receives the data uploaded by the equipment, according to the periods of time of data reporting, collecting all the data reported in the same period of time in the directory named by such period of time; (3), when storing the data, firstly buffering certain scale of reported data, when the buffered data achieves the set scale, intensively refreshing the buffered data to the file system of the server at once; (4), when querying the data, reading the corresponding files and screening out the query result according to the equipment identifier (id). The method provided by the invention can be used for reducing the IO (Input Output) pressure on the server caused by massive continuous data uploading of equipment, improving the amount of the uploading equipment which can be supported concurrently, and simultaneously achieving better balance on the read / write performance.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a massive data storage and retrieval method. Background technique [0002] With the popularization of the Internet and the widespread use of smart terminal devices, more and more terminal devices can now access the Internet and perform remote access, control and monitoring status through the network. How to effectively store the state information continuously and continuously uploaded by a large number of devices (tens of millions of levels) is a key issue for realizing online monitoring and historical state query of a large number of devices. For example, in the logistics industry, the location information flow of vehicles needs to be continuously uploaded to the server for storage. The uploaded data is uninterrupted for 24 hours. Due to the large number of vehicles, a large amount of data will be generated to be stored by the server. The IO load on the server storage system is h...

AI Technical Summary

Problems solved by technology

How to apply caching technology to massive data storage and retrieval is the key to solving the problem. For example, the patent application No. 201010142195.5 published by the State Intellectual Property Office is "a massive data information retrieval system and its method", including the terminal layer, The data service layer, data cache service layer and data storage service layer, because the caching technology involved in this patent is to cache the retrieval keywords and retrieval results, and the storage of massive data involves continuous reading and writing, and retrieval is only a process of reading. Thereby can't solve above-mentioned technical problem fully

Images