Multi-source streaming big data fusion convergence processing framework model implementation method

Abstract

The invention discloses a multi-source streaming big data fusion convergence processing framework model implementation method, which comprises a multi-source streaming data acquisition implementationmethod which comprises steps of realizing streaming data acquisition agent and multi-source streaming data acquisition service, providing an convergence metadata standard, and performing streaming multi-source data acquisition; and a multi-source streaming data fusion processing implementation method which comprises the steps of fusing and converging multi-source streaming data, providing a streaming processing engine based on Flink & Spark Stream; and according to the convergence metadata standard, configuring the collected streaming multi-source data fusion to carry out streaming data processing, processing the streaming multi-source data into convergence metadata meeting the convergence metadata standard requirement, carrying out regularized convergence metadata standard storage, and providing data for statistics, mining and analysis of the convergence data. According to the method, big data processing mainstream Flink and Sprakstream open source technical components are fused, andthe problems of multi-source data complementation, feature operator processing, convergence metadata standardization and streaming target data matching in multi-source data fusion convergence processing are solved.

Description

technical field [0001] The invention relates to the field of streaming big data aggregation processing, in particular to a method for realizing a framework model of multi-source streaming big data fusion aggregation processing. Background technique [0002] At present, the main focus of streaming data processing applications is in the field of distributed cloud computing. For example, a Chinese patent application document with the application number "CN201110450268.1" discloses a method for implementing a large-scale data continuous analysis system suitable for stream processing, which is suitable for a large-scale data continuous analysis system for stream processing, including The metadata management module is used to manage the metadata of data tables and databases; the query plan generation module is used to receive query requests and generate optimized query plans; the data import task generation module is used to receive data import requests and generate data import MR...

AI Technical Summary

Problems solved by technology

[0004] 1. The traditional incremental iterative processing of data is used to realize streaming data processing, which is not a real streaming processing method and is not suitable for the discovery of data targets with time-sensitive characteristics.

[0005] 2. There is no technical method to effectively implement stream data processing that satisfies high performance, as well as the analysis of the inherent laws of data processing.

[0006] 3. There is no technical solution for fusion and aggregation processing of multi-source data, and it is not suitable for fusion and aggregation processing of multi-source streaming data

[0007] 4. There is no user-definable technical method for multi-source data fusion aggregation processing

Images