Tight-coupling extensible big data interaction method

Abstract

The invention provides a tight-coupling extensible big data interaction method. By constructing a distributed tight-coupling client drive layer, on the basis of guaranteeing the consistency, single-point failure of the clients or server sides can be avoided, communication overhead among the clients is reduced, a system is enabled to have approximately-linear extensibility under scenes of emphasizing metadata query, and the demands of high online concurrency of interaction analysis of big data are met. According to the method, the read-write consistency of data can be guaranteed; even though the delay phenomenon is caused by pure read operation but the consistency of the sequence of a read-write version can be guaranteed; under the condition that a latest version is required to be read, the data synchronization process can be actively executed once; in addition, the method has good fault tolerance; as long as the number of failure nodes is smaller than a half, the influence on read-write data of other nodes is avoided; when the nodes reply, the synchronization can be realized by one-time read-write operation through the steps.

Description

technical field [0001] The present invention relates to the technical field of big data, and specifically relates to a tightly coupled and scalable big data interaction method. Background technique [0002] With the advent of the big data era, computing models and systems for data-intensive applications continue to emerge, such as the offline batch processing system MapReduce, the massive data high-concurrency processing system HBase, and the memory computing framework Spark and streaming. Storm, a traditional processing framework, and MPI, a traditional high-performance computing framework. In these big data processing modes, due to the introduction of new programming models, the learning cost is relatively high. Therefore, based on various big data processing systems, the requirements for interactive analysis modes and effects similar to traditional database applications are the most extensive. In interactive analysis, data is stored in the form of tables, and SQL stateme...

AI Technical Summary

Problems solved by technology

[0003] However, although the existing interactive analysis engine supports the table structure and SQL statement mode, and the underlying data system adopts a distributed architecture, the interactive analysis effect in practical applications is still very poor.

For example, Hive adopts the MapReduce engine and adopts the mode of strict synchronization and step-by-step materialization in each processing stage, and the processing delay is relatively large. Although Shark is based on the memory computing engine and optimizes processing performance through pipeline and intermediate result caching, because it uses the traditional Clinet / Sever mode, and perform SQL parsing, path planning, and metadata processing. The server side only supports single-point deployment, but restricts the effect of high-concurrency interactive processing.

Images