Massive data continuous analysis system suitable for stream processing

Abstract

The invention discloses a massive data continuous analysis system suitable for stream processing, which comprises a metadata management module, a query plan generation module, a data import task generation module, an increment processing module, an MR (MapReduce) message processing module and a database connection module, wherein the metadata management module is used for managing meta-information of data tables and databases; the query plan generation module is used for receiving a query request and generating an optimal query plan; the data import task generation module is used for receiving a data import request and generating a data import MR operation set; the increment processing module is used for incrementally committing data import and query operations to a Hadoop system in parallel; the MR message processing module is used for receiving a result of a Map or Reduce function of the Hadoop system and outputting the result to a Reduce end or the next operation; and the database connection module is used as an interface between the Hadoop system and the databases. According to the invention, the Hadoop system is used for organically organizing the databases in nodes and simultaneously executing data import and data query and a pipeline technology is used for improving the MR execution flow, so that the data query is executed in a continuous stream mode and the time of analyzing massive data is greatly shortened.

Description

technical field [0001] The invention belongs to the field of data analysis, and in particular relates to a large-scale data continuous analysis system suitable for stream processing. It is suitable for parallel analysis and calculation of large-scale data, and satisfies data analysis applications that require high query response time. Background technique [0002] With the advent of the era of big data, facing the problem of how to obtain valuable information from massive data, large-scale data analysis has become more and more important, and it also puts forward higher requirements for data analysis systems. The traditional method of using a single database management system (DBMS) for data analysis has been unable to support the ever-increasing data, nor can it meet the diverse and rapid data analysis requirements. [0003] The existing two main large-scale data analysis systems: Parallel DBMS (Parallel DBMS) and systems based on MapReduce (MR) framework have shortcomings...

AI Technical Summary

Problems solved by technology

However, most of the integration is only at the interface level, and few of them are integrated at the architectural level.

[0004] For the existing system that integrates the MR framework and database from the perspective of architecture, there are still problems of incomplete integration and failure to make full use of the advantages of the two, and lack of improvement to the existing architecture, which cannot well adapt to various , fast data analysis needs

Problems such as the long data import process and the batch design of MR have not been well resolved

Images