A distributed memory big data processing system and data processing method thereof

Abstract

The invention discloses a distributed memory big data processing system, which includes a proxy server and several data processing servers, the proxy server is connected to the client, the data processing server includes an aggregation module and several database modules connected to it, and all the aggregation modules are connected to the data Bus connection; during data processing, the interconnected aggregation modules form a tree structure, and the aggregation module as the root node is connected to the proxy server. At the same time, the data processing method of the system is also disclosed. The invention adopts a distributed method, can store massive data, and stores the data in memory, can achieve real-time response, and supports structured processing language at the same time.

Description

technical field [0001] The invention relates to a distributed memory big data processing system and a data processing method thereof, and the data belongs to the technical field of big data processing. Background technique [0002] The most common existing big data processing systems are Hadoop and LinDB. Hadoop is mainly composed of two parts, one is HDFS (distributed file system) for data storage, and the other is MapReduce (mapping and reduction) for data processing; the files managed by Hadoop are stored in several servers In the above, each block of each file is saved in HDFS with multiple backups (3 by default), and there is a dedicated service process in HDFS to maintain its file directory tree and the mapping relationship between its directory structure and the actual storage location of the file; MapReduce is a computing model and software architecture. It writes an application program running on Hadoop. It divides the large data to be processed by a job (Job) into...

AI Technical Summary

Problems solved by technology

Its overall design separates computing and storage, and is divided into computing layer LinProxy and storage layer LinStorage. The computing layer is mainly used for query analysis and intermediate combined re-aggregation calculations. The storage layer is mainly used for data writing. Data is classified according to time intervals. The new data is stored in the memory, and the older data will reduce the data time difference and save (only the maximum value, minimum value, average value, and number information of the original data are saved). The modified solution can achieve real-time query and comparison of monitoring data in the same period value, but does not support complex aggregate query operations

[0003] The above-mentioned Hadoop saves data files in the disk, and the response time of TB-level data processing (that is, data addition, deletion, modification, and query) is slow; although LinDB saves data in memory and can respond in real time, the data capacity is small and the processing method Single, does not support structured processing language; therefore, there is an urgent need for a system with large data capacity, real-time response, and support for structured processing language

Images