A data acquisition management system and method

Abstract

The invention relates to a data acquisition management system and method. The system comprises system management, code management, data quality control, resource database management, data source management, ETL parameter configuration, ETL operation management, ETL operation log, monitoring node configuration, operation monitoring and analysis. The system is connected with Kettle C / S data acquisition process configuration; The method comprises the following steps of: in the data acquisition process, automatically generating the table structure involved in the mechanism database needed for platform construction, automatically collecting the data within the scope of configuration rules, and automatically verifying the generated data quality inspection results; The invention solves the problem of too many roles of personnel existing in the prior art, The invention realizes the dynamic configuration of the data standardization collection, data convergence, data cleaning, code transcoding,data quality checking and other processes of the big data platform construction, and has the advantages of smooth operation, controllable data quality, easy implementation and maintenance, and high efficiency and stability of data collection.

Description

technical field [0001] The invention relates to the field of medical technology, in particular to a data collection and management system and method. Background technique [0002] At present, all walks of life are building big data platforms, and the construction of big data platforms requires the collection of various business data in the industry. The existing methods of collecting business data are: [0003] 1) Export data files in different places, and then copy them to the platform for import and continue data cleaning. In this method, the data is not timely, the data file is damaged or the data is lost during the export process, and the processing is troublesome and time-consuming. [0004] 2) Data-driven approach. The subordinate industry organization provides a server, and establishes a platform data underlying structure on the server. The subordinate organization directly pushes and writes the data, and then the platform fetches the pushed data to the central end...

AI Technical Summary

Problems solved by technology

[0009] 1) There are too many personnel roles in the data operation process, and it is inconvenient to form a unified understanding of the overall process

[0010] 2) There are many timing task rules in the data operation process, and they are set in different systems or tools and different networks, so the connection between timing tasks is inconvenient

[0011] 3) If there is a problem in the whole process, all operators need to participate in the troubleshooting, there are many related parties, and a lot of manpower is invested, and it may cause problems to be blamed, which is not conducive to rapid analysis, positioning, and problem solving

[0012] 4) For operations such as data interface upgrades and repeated data pulls, the existing process will result in long implementation cycles, multi-manpower and material resources investment, multi-party, multiple coordination and communication, and data quality cannot be guaranteed

Images