A Storm-based big data preprocessing method and system for railway geological disaster monitoring

Abstract

The invention discloses a storm-based method and system for preprocessing big data of railway geological disaster monitoring, including: providing a parallel sub-calculation module for each sensor data of each monitoring point based on the open-source stream computing storm framework, and each sub-computation The module realizes ETL processing of railway geological disaster monitoring big data, realizes real-time parallel preprocessing of multiple sensor data at different monitoring points, effectively improves data processing efficiency and data cleaning quality; effectively solves the problem of low efficiency of serial processing in traditional methods , can not meet the processing requirements of monitoring big data, can meet the real-time, timeliness, high precision and other requirements of railway geological disaster monitoring data preprocessing, and is suitable for complex railway geological disaster monitoring big data real-time analysis and early warning application scenarios; at the same time The storm-based ELT processing method provided by the present invention is also applicable to other complex projects with multiple project monitoring points and various monitoring data.

Description

technical field [0001] The invention relates to the application field of equipment monitoring and analysis, in particular to a storm-based method and system for preprocessing big data of railway geological disaster monitoring. Background technique [0002] In the application of big data analysis, the data quality of data preprocessing is a key factor related to the credibility of data analysis and mining. Similarly, the high efficiency and real-time performance of data preprocessing is also very important for many applications. In the application of geological disaster monitoring and early warning along the railway, a large amount of monitoring data collected by sensors reaches the monitoring processing platform in real time in a fast manner. How to realize fast data extraction, data conversion, and data cleaning on the monitoring processing platform faces many challenges. Quality and fast data preprocessing is necessary. The big data of railway geological disaster monitor...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to overcome the traditional ETL data processing method existing in the prior art, which can only accurately analyze a single monitoring data, the serial data processing efficiency is low, the data cleaning quality is low, resulting in low monitoring and early warning accuracy, which is not applicable Due to the defects of complex railway geological disaster monitoring big data processing, a storm-based railway geological disaster monitoring big data preprocessing method and system are provided. This invention combines the characteristics of distributed stream computing processing technology, and proposes an open source stream computing framework. The big data preprocessing method for railway geological disaster monitoring uses the storm flow computing framework and distributed parallel computing technology to carry out real-time ETL preprocessing for massive big data of geological disaster monitoring along the railway, and improves the traditional big data ETL method for geological disaster monitoring in terms of data processing efficiency, Limitations in the quality of data cleaning and the like; the method can not only realize ETL rapid preprocessing of railway engineering geological disaster monitoring big data, but also provide high-quality basic data for monitoring and early warning analysis; the method of the present invention is also applicable to other railway engineering fields ETL preprocessing for big data analysis of multiple engineering monitoring points

Images