Monitoring data early-warning system based on Spark flow-type clustering

Abstract

The invention discloses a monitoring data early-warning system based on Spark flow-type clustering, and the system employs a method which comprises the steps: collecting environment monitoring data inreal time through a sensor cluster, and transmitting the collected data to a cloud platform through GPRS; Writing the mass data into an HBase database in real time through Spark Stream, achieving a flow-type K-Means clustering method through a Spark machine learning library MLlib, dividing monitoring data into three types in an unsupervised manner: normal environment data, strong feature environment data, weak feature environment data, carrying out the clustering analysis of the collected data through the flow-type K-Means clustering method, and judging whether a prediction result accords with the clustering result or not; starting a client computer buzzer and sending an alarm short message to a cellphone client when abnormality data is found during clustering analysis. The method can achieve the effective storage and reliable transmission of the mass meteorological data, and greatly improves the reasonability and accuracy of the analysis and processing of the meteorological data.

Description

technical field [0001] The invention belongs to the field of the Internet of Things and big data, and in particular relates to a monitoring data early warning system of Spark streaming clustering. Background technique [0002] The current monitoring and early warning system mainly uses GPRS data service, WEB-GIS for data collection, and local area network data transmission technology. The monitoring data uses small and medium-sized relational databases for data storage. For data analysis and decision-making capabilities, effective analysis and classification judgment of monitoring data cannot be realized in real time. Contents of the invention [0003] The purpose of the present invention is to provide a monitoring data early warning system based on Spark streaming clustering. In order to achieve real-time monitoring of environmental monitoring data, model clustering and abnormal data early warning mechanism. [0004] The technical solution to realize the object of the p...

AI Technical Summary

Problems solved by technology

[0002] The current monitoring and early warning system mainly uses GPRS data service, WEB-GIS for data collection, and local area network data transmission technology. The monitoring data uses small and medium-sized relational databases for data storage. For data analysis and decision-making capabilities, effective analysis and classification judgment of monitoring data cannot be realized in real time

Images