Partitioned connection method oriented to mixed type big data processing systems

Abstract

The invention discloses a partitioned connection method oriented to mixed type big data processing systems. The partitioned connection method oriented to mixed type big data processing systems is capable of satisfying the transactional analysis business application demands of the industrial big data in allusion to different processing systems and greatly improving the property of the analysis through partition, coprocessr and mapjoin, and can be further applied to the transactional analysis of join-based grouping, counting and sorting. According to the partitioned connection method oriented to mixed type big data processing systems, the size of the data joining in the transmission, cache and join processes is decreased through determining the Hive query partition; by sufficiently utilizing the advantages of the distributed structure, the cache processes of all the nodes are executed in parallel; through caching data at each node, the join execution efficiency can be accelerated; and the data size and the node amount of the HBase table can be extended as required.

Description

technical field [0001] The present invention relates to the technical field of big data, and specifically relates to a partition connection method for hybrid big data processing systems. Background technique [0002] In response to the industry's big data business application requirements, computing frameworks and systems for data-intensive applications continue to emerge, and these systems only provide solutions for their respective problem domains. In order to cope with the increasingly complex business needs of the industry, it is necessary to use multiple processing architectures in large-scale clusters or data centers to store and process massive amounts of data. Therefore, a hybrid big data processing system has emerged, which integrates multiple processing modes such as batch processing, memory computing, stream processing, and NoSQL database, such as the YARN architecture, to meet the real-time processing, interactive processing, efficient retrieval, and in-depth pro...

AI Technical Summary

Problems solved by technology

However, the interactive analysis engine in the existing hybrid big data architecture is only for a single big data system, and cannot realize cross-system data processing

For example, the current data in Hive and HBase cannot be directly associated. The usual practice is to perform a data migration in a single system of Hive or HBase. The data redundancy and transmission delay caused by a large amount of data are intolerable of

Images