A method for decentralized big data management and retrieval

Abstract

The invention relates to a distributed massive data management and retrieval method, which comprises the following steps: an application program server processes Beidou space data, a management server stores the processed data in a file server according to a configuration in a distributed manner, and records storage locations, a Web server which provides service for access of an external network sends an external access request to the application program server, and the application program server acquires the distributed file data through the management server, processes the data according to the access request, and returns the data to the Web server, wherein the data is divided into two parts, one part is basic data and the other part is business data; after a system runs, the system reads the data into a memory, and directly operates a data file by virtue of a buffer of the memory. According to the method, the time-space distribution of fishing vessels can be statistically queried accurately and directly.

Description

technical field [0001] The invention relates to the technical field of big data management, in particular to a distributed big data management and retrieval method, especially the distributed storage management of a large number of Beidou ship position spatio-temporal data, as well as fast query and statistics. Background technique [0002] Massive business data application systems need to query a large amount of relevant data during operation, so the storage and query of business data has become an important part of the application system. Due to the large amount of data, for the convenience of maintenance and management, the business data of the application system is usually stored in the database, and the application system queries the database by using SQL statements at runtime. Although this method is simple to implement, due to factors such as the performance of the database itself and the consumption of network data transmission, it will affect the speed of applicatio...

AI Technical Summary

Problems solved by technology

Although this method is simple to implement, due to factors such as the performance of the database itself and the consumption of network data transmission, it will affect the speed of application processing query data, thereby reducing the performance of the application system. Therefore, the method of directly accessing the database is more suitable for data processing. A system with a small volume and low requirements on the processing speed of the application system

In addition, database systems such as DB2, Oracle, and SQL Server bypass the file system and manage disk blocks by themselves, which improves the performance of the database, but the management of the database becomes more complicated, and the cost of these commercial databases is relatively high.

Images