Big data processing system and method

Abstract

The invention discloses a big data processing system and method. The system comprises: a service interface which provides at least one external calling mode and is suitable for receiving a query statement in a specific language format, which is input by utilizing any external calling mode; an analysis module which is suitable for performing grammar analysis and verification on the query statementto generate a logic query plan; a routing module which is suitable for determining a calculation engine and / or a storage engine corresponding to the logic query plan according to the logic query plan,and routing the query statement to the calculation engine and / or the storage engine; and the plurality of computing engines and the plurality of storage engines which are suitable for executing corresponding query processing according to the query statement routed by the routing module, and obtaining and outputting a query result. By adopting the scheme, the user can process the big data only through the input query statement in the specific language format, so that the service logic of the user is decoupled from the calculation engine and the storage engine, the learning cost of the user isreduced, and the user experience is improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a big data processing system and method. Background technique [0002] With the continuous development of science and technology and society, all kinds of data are growing in a blowout style, and at the same time, a large number of big data platforms are constantly emerging, and people can realize the processing of massive data through big data platforms. [0003] The current big data platform still requires users to select the computing engine and / or storage engine required for data query, and then compile the corresponding execution code according to the engine characteristics and grammatical rules of the selected computing engine and / or storage engine, so as to realize Queries on data. [0004] However, using the existing big data platform, users need to learn a lot of computing engine and storage engine knowledge, which greatly increases the user's learning costs and reduc...

AI Technical Summary

Problems solved by technology

[0004] However, using the existing big data platform, users need to learn a lot of computing engine and storage engine knowledge, which greatly increases the user's learning costs and reduces user experience; moreover, with the rapid development of technology today, the iterative update of computing engines and storage engines is relatively fast , while the existing big data platform increases the learning cost for users, it is also prone to situations where the selected computing engine or storage engine does not match the actual business logic due to the lack of knowledge of the computing engine or storage engine by the user. Thereby reducing the disadvantages of data processing efficiency; in addition, because the existing big data platform data processing logic is too closely coupled with the computing engine or storage engine, when the computing engine or storage engine is replaced, the code needs to be recompiled, thereby improving business maintenance cost

Images