Distributed memory big data processing system and data processing method thereof

A big data processing and data processing technology, applied in the field of big data processing, can solve the problems of small data capacity, slow response time of TB-level data processing, and no support for structured processing languages, and achieve the effect of balanced burden

CN108920542AActive Publication Date: 2018-11-30西安炫月云网络科技有限公司
8 Cites 4 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2018-11-30

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a distributed memory big data processing system, which comprises a proxy server and a plurality of data processing servers. The proxy server is connected with a client, each data processing server comprises an aggregation module and a plurality of database modules connected with the aggregation module, and all aggregation modules are connected with a data bus. In data processing, connected aggregation modules form a tree structures, and the aggregation modules serving as root nodes are connected with the proxy server. The invention further discloses a data processing method of the system. By a distributed mode, mass data storage can be realized; by storage of data in a memory, real-time response can be realized, and structural processing languages are supported.
Need to check novelty before this filing date? Find Prior Art

Description

Technical field

[0001] The invention relates to a distributed memory big data processing system and a data processing method thereof, and the data belongs to the technical field of big data processing. Background technique

[0002] The most common big data processing systems available are Hadoop and LinDB. Hadoop is mainly composed of two parts, one is HDFS (distributed file system) for data storage, and the other is MapReduce (mapping and reduction) for data processing; the files managed by Hadoop are sliced ​​and stored on several servers Above, each slice of each file saves multiple backups in HDFS (default 3). There is a dedicated service process in HDFS to maintain the file directory tree and the mapping relationship between its directory structure and the actual storage location of the file; MapReduce is a computing model and software architecture. It writes applications that run on Hadoop. It divides the big data to be processed in a job (Job) into many data blocks, and e...

Examples

Embodiment Construction

[0031] The present invention will be further described below in conjunction with the drawings. The following embodiments are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

[0032] Such as figure 1 As shown, a distributed memory big data processing system includes a proxy server and several data processing servers.

[0033] The proxy server is connected to the client. The data processing server includes an aggregation module and several database modules connected to it. All aggregation modules are connected to the data bus. All servers transfer data through RPC calls, and the consistency of configuration is ensured through zookeeper. During data processing, interconnected aggregation modules form a tree structure, such as figure 2 As shown, the aggregation module as the root node is connected to the proxy server.

[0034] The functions of each part are as follows:

[00...