Distributed data analysis processing method based on memory computation

Abstract

The invention provides a distributed data analysis processing method based on memory computation, comprising the following steps of (1) providing an SQL-like (Structured Query Language) parser which parses an introduced query analysis text into corresponding logical plans and optimizes the logical plans preliminarily; (2) providing a task converter which converts the logical plans generated by the SQL-like parser into a computation expression, including a plurality of defined conversion categories, which can be recognized by a big data memory computational model; (3) providing a query optimizer which converts the introduced expression that can be recognized by the memory computational model into the logical plans, optimizes the logical plans and then converts into physical execution plans. Due to lack of data query and analysis processing in the conventional big data processing, the invention provides the distributed data analysis processing method based on memory computation and inherits the advantages of the memory computational model in the aspect of data processing so that the programming language of data query and analysis is simple.

Description

technical field [0001] This article relates to a field of big data analysis technology, specifically a distributed data analysis and processing method based on the memory computing framework. Background technique [0002] With the advent of the era of cloud computing, the emerging term "big data" has attracted more and more attention; the so-called big data refers to data that cannot be captured, managed and processed with conventional software tools within an affordable time frame. Data collection; the meaning of big data does not simply refer to mastering huge data information, but how to extract the key information contained in a large amount of data, that is, how to process and analyze big data; of course, for distributed data mining of massive data, it is necessary Relying on distributed processing of cloud computing, distributed database and cloud storage, virtualization technology, etc. [0003] With the rapid development of the Internet and the rapid expansion of in...

AI Technical Summary

Problems solved by technology

[0003] With the rapid development of the Internet and the rapid expansion of informatization data, there are more and more difficulties in the analysis and processing of massive data. In addition, the continuous development of cloud computing platforms has also brought new challenges and opportunities for data processing and analysis. Traditional data Analytical methods are increasingly unable to meet the requirements; big data is an emerging industry, and the key to gaining benefits lies in the "value-added" of data through "processing" of data, and the rate of "processing" will inevitably determine the "value-added" of benefits. " space, so how to quickly query and analyze massive data becomes particularly important

[0004] The analysis and processing of data by the traditional data analysis platform is not real-time. This is because the traditional data computing framework has the disadvantage of having to read and store. When the amount of data is too large, the processing is often slow, resulting in a large amount of delay; while the new computing The framework is based on the memory computing method, which not only inherits the distributed processing framework of the traditional computing framework, can automatically optimize the computing process, etc., but also makes up for the shortcomings of the traditional computing framework that do not have real-time performance, which greatly improves the data processing speed; therefore, we need a A platform for data analysis and processing based on a new in-memory computing framework, making big data processing faster and more convenient

Images