Distributed memory big data processing system and data processing method thereof

Abstract

The invention discloses a distributed memory big data processing system, which comprises a proxy server and a plurality of data processing servers. The proxy server is connected with a client, each data processing server comprises an aggregation module and a plurality of database modules connected with the aggregation module, and all aggregation modules are connected with a data bus. In data processing, connected aggregation modules form a tree structures, and the aggregation modules serving as root nodes are connected with the proxy server. The invention further discloses a data processing method of the system. By a distributed mode, mass data storage can be realized; by storage of data in a memory, real-time response can be realized, and structural processing languages are supported.

Description

Technical field [0001] The invention relates to a distributed memory big data processing system and a data processing method thereof, and the data belongs to the technical field of big data processing. Background technique [0002] The most common big data processing systems available are Hadoop and LinDB. Hadoop is mainly composed of two parts, one is HDFS (distributed file system) for data storage, and the other is MapReduce (mapping and reduction) for data processing; the files managed by Hadoop are sliced ​​and stored on several servers Above, each slice of each file saves multiple backups in HDFS (default 3). There is a dedicated service process in HDFS to maintain the file directory tree and the mapping relationship between its directory structure and the actual storage location of the file; MapReduce is a computing model and software architecture. It writes applications that run on Hadoop. It divides the big data to be processed in a job (Job) into many data blocks, and e...

AI Technical Summary

Problems solved by technology

Its overall design separates computing and storage, and is divided into computing layer LinProxy and storage layer LinStorage. The computing layer is mainly used for query analysis and intermediate combined re-aggregation calculations. The storage layer is mainly used for data writing. Data is classified according to time intervals. The new data is stored in the memory, and the older data will reduce the data time difference and save (only the maximum value, minimum value, average value, and number information of the original data are saved), and the modified scheme can achieve real-time query and comparison of monitoring data in the same period value, but does not support complex aggregate query operations

[0003] The above-mentioned Hadoop saves data files on disk, and the response time of terabyte-level data processing (that is, data addition, deletion, modification, and query) is slow; although LinDB stores data in memory and can respond in real time, the data capacity is small and the processing method Single, does not support structured processing language; therefore, there is an urgent need for a system with large data capacity, real-time response, and support for structured processing language

Images