Big data acquisition method

Abstract

The invention discloses a big data acquisition method, which comprises the following steps of S1, uniformly processing different types of data based on a visual process arrangement platform; S11, constructing a design state platform; S12, customizing different access components for different types of data; S13, presetting an access assembly; S2, uniformly arranging data reconciliation and data quality detection public flows based on a visual flow arrangement platform; S21, arranging a data reconciliation assembly and a data reconciliation process at a node capable of actively acquiring sourcedata and destination data; and S22, arranging a data quality detection assembly and process at a node capable of actively pulling data. According to the invention, the difference of data source typesis shielded by a pre-acquisition processor, and then the data can be processed by a unified real-time stream engine, so that the processing logic is relatively clear and simple; and meanwhile, publicprocesses such as data reconciliation and data quality detection are conveniently and uniformly arranged, and the quality and efficiency of the whole system are improved.

Description

technical field [0001] The present invention relates to the technical field of data collection, and more specifically to a data collection method for big data. Background technique [0002] Data collection is an important part of the big data governance platform. The big data governance platform faces a wide variety of data sources and diverse forms. When collecting data, it is necessary to monitor and receive high-speed real-time data streams, and also need to actively pull massive static data; it is necessary to process regular structured data , also need to deal with a large amount of semi-structured and unstructured data. [0003] The current data collection methods on the market have the following deficiencies when accessing: 1) The collection and processing processes are organized separately according to different scenarios, and unified data processing cannot be performed, and the processing logic is relatively complicated; 2) It is not conducive to data reconciliatio...

AI Technical Summary

Problems solved by technology

[0004] The technical problem to be solved in the present invention is to provide a data collection method for big data, so as to solve the problems that the current data collection methods on the market cannot perform unified data processing, and are not conducive to the arrangement of public processes such as data reconciliation and data quality monitoring. , in order to realize the ability to process data through a unified real-time streaming engine, making the processing logic clear and simple, so as to facilitate the unified arrangement of public processes such as data reconciliation and data quality monitoring

Images